koobas.hobune.stream
How do I remove floating point errors when setting rotations with an editor script!?
Ok, so I have an editor script I am building for laying out roads in the unity editor. One of the things it does is rotate the road segments 90 degrees to the right or left. EVERY SINGLE METHOD of doing this leaves a floating-point error, resulting in the rotation of the object being 90.00001 rather than 90.
Now I understand that's theoretically a limitation of floating-point values and all that except:
A: It is actually noticeable, you can see a pixel line where the roads don't line up
and more importantly
B: It can be fixed by editing the rotation manually in the transform inspector
So unity IS capable of getting around the floating-point error, but only through the inspector interface. All attempts to set the value via code result in the same stupid floating point error.
I NEED to remove the floating-point error because the other option is to have our level designer manually fix the rotation for EVERY. SINGLE. ROAD.
I have tried the following:
//The obvious first attempt
transform.eulerAngles = new Vector3(0, 90, 0);
//Set the Quaterinion rotation directly
transform.rotation = new Quaternion(0, 0.7f, 0, 0.7f);
//Set the quaterninon rotation directly using values reported from debug logs after manually setting rotation to exactly 90
objs[i].transform.rotation = new Quaternion(0, 0.7071068f, 0, 0.7071068f);
//Set the Quaterinion rotation directly but add a bunch of extra zeros to try to move the decimal place out to somewhere it won't cause problems
transform.rotation = new Quaternion(0.00000000f, 0.70000000f, 0.00000000f, 0.70000000f);
//Set based on angle axis
transform.rotation = Quaternion.AngleAxis((int)90, new Vector3(0, 1, 0));
//Set by modifying transform forward
transform.forward = new Vector3(1,0,0);
//use rotate
transform.Rotate(0, 90, 0);
What I assume is happening is that every method to set rotation ultimately goes through the same flawed code path, except for the transform inspector. So what I think MIGHT work is somehow having my script access the transform inspector rather than the transform values themselves but I am not sure if unity will even allow me to do that. So far I haven't found a way to access the inspector UI through an editor script. Because why would you need to when your editor script could just set values directly, etc...
Any help anyone can give me would be greatly appreciated.
Have you tried to use floats instead of integers when you create a vector?
new Vector3(0, 90f, 0);
This should be enough, but you could do it even more explicitly for the 0's:
new Vector3(0f, 90f, 0f);
or
new Vector3(0.0, 90.0, 0.0);
The thing is Unity's Vector constructor takes float arguments, and when you give integers to it, C# automatically performs an implicit conversion to convert integers to floats, which is known to have some precision related issues.
Hurm. no luck. Tried the follwoing
transform.eulerAngles = new Vector3(0, 90f, 0);
transform.eulerAngles = new Vector3(0f, 90f, 0f);
They both resulted in the same floating point error.
This won't compile as it doesn't realize they are floats without the f at the end.
transform.eulerAngles = new Vector3(0.0, 90.0, 0.0);
Ok, I am going to tentatively answer this myself but I should preface this with I have NO IDEA why this works. Still, it does appear to solve the problem so I'll post it here.
I was following this tutorial to learn how to create a custom inspector for an entirely different issue: https://naplandgames.com/blog/2016/08/27/unity-3d-tutorial-custom-transform-inspector/
As far as I can tell it's not doing anything that should correct the issue. And yet it does. When using this custom inspector it does three noticeable things.
A: It is way more accurate. Showing the full scientific notation where applicable (eg 9.334668e-06).
B: It will auto flip negative values. (e.g. enter -90, it will be converted to 270)
C: It will Ș̴͖́́i̶͓̓̑ͅl̶͕̖̔̓ė̸̩͝n̵͎̳͌̃c̵̜̈ḙ̶̐ t̶̖͛̾h̶̨̙͐ẽ̷̩ e̴̟͒͠l̵̡͊͝d̶͔̮̾r̶̯̘̾̆i̷̹͕͘t̴͓̭͆́c̵̲͂̎ͅh̶̳̔ g̴̰̅͘ǒ̷̼͌d̴͙̈́s̶̜͕͂ t̴̜̞̅h̷̘̫̏̕a̵̧̦̍͘ṱ̸̓͜͠ c̶̨̝̆͝o̶̥͒̿ń̶̪̌t̶̰͒r̵̝͛͂o̵͓̔͌l̶̼̘̔͝ u̴̳̦̓̑n̵͙͔͋͝ī̵̫͛t̸͖̯̉̊i̴̡͚̎ė̵͉s̸͓͂͜ f̴̠̺͒l̴̺͊́ö̸̫́ą̷̆t̷͖̆i̵̝͌̃n̴̲̚g̷̙̏ p̶͍̓͑ȏ̶̩i̸̯̳͗̌n̷̼͇̾t̸̫̜̋ i̴̫̍s̷͍͎̓s̴͔̀ü̸͓é̸̗̾.̶̝̅
The main difference that the custom inspector does that you don't is that the custom inspector correctly sets the local rotation / localEulerAngles while you are setting the worldspace rotation which gets converted internally into local space rotation since the local space rotation is the one that actually specifies where and how this object is oriented in space.
