koobas.hobune.stream
JsonUtility serializes floats with way too many digits
I need to serialize a Vector3 to Json. For this i want to use the build in JsonUtility. But JsonUtility converts a Vector3 to floats with a lot of decimal places. I know that this is the way a float works but it is not practical for serialization to a file. The files will become unnecessarily large because of all the digits.
Here an example:
Debug.Log(JsonUtility.ToJson(new Vector3(0.24f, 0.5f, 0.75f)));
will output:
{"x":0.23999999463558198,"y":0.5,"z":0.75}
can i do something to make this more efficient? I used Json Net before where this was no problem. However Json Net is a lot slower than JsonUtility (see here) and i would like to use as little 3rd party libraries as possible.
You've found the float problem.
You could try a double, which may seem counter intuitive, and doesn't always solve the problem, but here's what's happening to you:
float isn't accurate. There is a limited amount of storage, and it is typically squeezed into 32 bits. It is fashioned with a two part design, a mantissa and exponent. It is the binary version of scientific notation (like 2.4 e-1). However, the mantissa is limited to fewer than 32 bits. The exponent takes up some of those bits. In the standard IEEE format for this (which I think C# is using here), this means you have 24 bits for the precision (a base 2 version of the digits involved), and 8 bits for the exponent (expressing the power of 2 those digits should be shifted), and both are signed (so you have negative exponents for shifts toward very small values).
As a result of this encoding, the representation isn't exactly. .24 becomes a close approximation that when converted into base 10 looks like a garbled mess.
It is, however, normal, and even internally, the number .24 may indeed be "known" to your C# code as the value JSON is showing, and not actually .24f.
Doubles have more precision, and therefore have a chance to be more accurate, but in practice all that really means is that the base 10 representations are different (maybe .24 looks exact, but .242 doesn't). There are various formats, some considered non-standard.
This goes all the way down to the CPU, as it uses at least some standard for these representations which may be the same used by the language, and thus fundamentally there's nothing you can actually do if you're going to use floats.
You could, however, use other numbering systems. Fixed point values could be a choice, where an integer (possibly 64 bits) represents units such that you have a fixed number of decimal values that never "float" like this. For example, you could decide you want 4 digits of precision, so you'd consider all integers 1000 units, so that 240 is actually .24.
Other than that, this is a fundamental issue for which there are a wide range of solutions you'll find all over the web, and can choose among many that might better suit your preference for a particular application.
Or, like many of us, just accept that's how it works and forget about it.
Thank you for the explaination. It looks like i have to switch to another serializer. Just accepting is pretty bad i would say. if a float that has 3 digits is converted to 18 characters in the file it is a big deal. The file will use about 5 times more storage than it needs. I actually just need to serialize a float like ToString("F4") does. I realy have doubt in the JsonUtility if a basic float sometimes needs 6 times more space just for a super tiny accuracy difference. It is not possible to round floats in the serialized Json with JsonUtiliy! it's pretty stupid in my opinion!
As far as I can see, the built in JSON has no configuration options for itself. That said, you should probably have a look at custom serialization in Unity, which might solve your problem.
I've never used it, but there are open source libraries like:
It appears to have options for serialization control, maybe a bit too granular, but it might do what you need. Incorporating it within Unity may be a squeeze, and perhaps there are others, but JSON is among the very simplest I/O means, and you might just open a file and write out text as you prefer it (rounding as required), in the correct output format. You could still use JSON built in to read it if you get the output formatting correct, which is all but trivial.
That said, opening a file and writing data isn't entirely platform independent - especially where iOS is concerned, so target hardware/os invades the topic.
You're right, though Unity's JSON serializer (as well as most others) use double values by default. The number above has about 17 significant decimal digits (53 bits) which matches the range of a double value. A float / single value has only about 7 significant decimal digits (24 bits)
Using double is actually suggested by the JSON standard
Since software that implementsIEEE 754-2008 binary64 (double precision) numbers [IEEE754] isgenerally available and widely used, good interoperability can beachieved by implementations that expect no more precision or rangethan these provide, in the sense that implementations willapproximate JSON numbers within the expected precision.
However there's no real general valid precision limits that have to be applied to json numbers. They could be as long as you want. Though for compatibility you should never assume a higher precision than what double provides. If you need more precision it's adviced to store the value as string. If you want to play around with float or double values and their binary representation, have a look at my floating point format editor window for Unity.
As for the speed differences between the different serializers: Unity did implement their json serializer in native code. So obviously no managed garbage will be allocated. Unity's serializer is quite limited in what it can serialize. In addition, as you said, it doesn't have any configurations or settings. This is what it makes so fast. Not having to deal with all possible cases and having to incorporate various settings while serializing makes it fast. It's the typical trade-off between features and speed. You have to decide what you need or how you can workaround what you need. The ISerializationCallbackReceiver could be used to simply replace the float with an int and store it as a multiple of 1000 as you said above.
Just for completeness: If you don't care about how the value is stored in the json data and all you need is to store and read the data from Unity you can store the value either as integer or as string. This can be easily done with Unity's ISerializationCallbackReceiver interface. You could even wrap the serialization into a seperate struct / class like this:
public struct FloatWrapper : ISerializationCallbackReceiver
{
[System.NonSerialized]
public float someValue;
[SerializeField]
private int _someValue;
public void OnAfterDeserialize()
{
someValue = _someValue / 1000f;
}
public void OnBeforeSerialize()
{
_someValue = (int)(someValue * 1000);
}
}
In your code you just use the public float value. However when serialized we actually store the value as integer so you basically preserve 3 decimal digits behind the decimal point. So a value like "25.12" would be stored as 25120. Another way would be to use a string as serialized value and format the float as you like using the serialization callbacks. However in this case the value in the json data would be a string and not a json number
// [ .... ]
private string _someValue;
public void OnAfterDeserialize()
{
float.TryParse(_someValue, out someValue);
}
public void OnBeforeSerialize()
{
_someValue = someValue.ToString("UseYourDesiredFormat");
}
// [ .... ]
Of course you have to take care of that the deserialization of the string value into a float works. Of course as a string value you have two additional quote characters in your json data.
