koobas.hobune.stream
Why do those two calculations return different values?
Hi,
I want to display the first three numbers after a point of a float value. Example: float 61.001 => 1
I have done this (using c#):
float value = 61.001; float temp = (value*1000); int retVal = ((int)temp)%1000;
retVal has value 1 -> This is correct
but when I am not using the temp var:
float value = 61.001; int retVal = ((int)(value*1000))%1000;
retVal has value 0, what is not correct.
I really wonder why those two calculations return different values. In my three or four tests done, the second implementation seems to return 1 less than the other. (ie. 58 instead of 59)
This cant be caused by float rounding, can it? Is there a optimization that causes this difference?
Thanks!
It doesn't really make sense to get the "decimal places" of a float... if you want accurate decimal precision you should use the decimal type.
Did you try with an other initial value than 61.001?
Personally, I would never use float for such precision. float points are just not designed for precision. As tanoshimi pointed out, you should use decimal type if you want extreme precision.
I don't think you can ever be sure of the optimisations and approximations that will occur with floats calculations. And thus, you should never expect an exact result.
Also, just adding, I'd say that 61.001 is a double. 61.001f is a float. So your first assignation is actually a cast from double to float, introducing a first possibility to wreck your precision.
If you want to print float rounding to 3 decimal points you should call value.ToString("0.000");
[1]: http://msdn.microsoft.com/en-us/library/dwhawy9k%28v=vs.110%29.aspx
[2]: http://msdn.microsoft.com/en-us/library/0c899ak8%28v=vs.110%29.aspx
I dont want to round a float. I want to get the first three decimal places
I turned retVal to a float and took out the parse and I get the value
0.9994507
Without the %1000 it returns 61001, spot on.
The error itself is in the (non) trucation of tiny values from value*1000. If performed all in the same calculation the tiny decimals seem to be retained. When you separate it with the temp variable the value is 61001 on the dot, so returns 1 as expected.
So, yes, float error is the culprit, but int rounding is what gives the 0 as even 0.999999 float = 0 int. Always rounds down.
It's about double precision and rounding to floats - Here is the answer: http://stackoverflow.com/questions/8911440/strange-behavior-when-casting-a-float-to-int-in-c-sharp
Because 61.001f * 1000 returns a double at first and is then stored in a variable rounded to the nearest float which is
61000.9999... to float = 61001.000...f
In the other case (int)(61.001f * 1000) it returns a double at first and is then casted to an int directly which is
61000.9999... to int = 61000
It will work without a temp variable if you cast the inner term to a float first:
int retVal = ((int)((float)(value*1000)))%1000;
