koobas.hobune.stream
Unity's Serialization seems to have a maximum depth of 7. How can I serialize a complex data structure with depths > 7?
I have a Trie (N-tree) data structure with ~100k total elements (large). I need to pre-compute the Trie because it's very slow (~1.5 min).
I marked it as Serializable, and placed it on a Prefab and pre-computed the data. This worked great! Unity loaded it with very little hiccup and I had all my data in a Trie ready to go!
Then I realized all of my elements of depth 7+ were missing when I used the serialized Trie. It appears Unity Serialization has a max depth cutoff.
So my question is, what is the best way to serialize a complex data structure of high depths in a way that provides very fast load/deserialization time?
My attempted solutions:
Use BinaryFormatter to write my own serialized binary data file to disc, and load it back during the game. This proved horrendously slow and created a huge file (142MB).
Forget serializing and speed up the tree creation by parallelizing it. This still causes a 20-30 second delay on the iPad, vs the unnoticable delay using Unity's prefab serialization.
Anyone know what Unity is using to serialize and deserialize data? It's very fast at loading in comparison to anything I've tried.
Any help would be greatly appreciated. Thanks!! :D
you may be able to work around this by storing each branch's depth in that branch - if the depth is 7 or less, data continues in this object. if the depth is seven, ins$$anonymous$$d of creating more serializable lists or whatever, create links to child gameObjects which then continue the trie in their own data.
i don't know of an answer to this, unfortunately. every time i've seen this question come up it has simply not been solved. :(
have you checked out UnitySerializer by whydoidoit, the most senior unity dev. Almost everyone uses it. (I'm afraid I do NOT actually know if it handles the "7" issue you mention, sorry. But it's the starting point for all serious serialization woes, saved us a fortune.)
It'll serialize 7 deep+ no problem. Unfortunately using it for just one property involves knowing when to save it and then storing the result in a byte array for Unity to save with your level. It's not that hard and I use it all the time for Dictionaries/complex things etc.
US should be smaller than BinaryFormatter output as it uses a number of levels of compression and default value avoidance. Sounds like you Trie is big though.
i had utterly no idea "Trie" (it is pronounced as in re*trie*val) was a term for a prefix tree! awesome.
http://en.wikipedia.org/wiki/Trie
"The term trie comes from retrieval. This term was coined by Edward Fredkin, who pronounces it "tree" as in the word retrieval."
http://en.wikipedia.org/wiki/Edward_Fredkin
!! "digital philosophy!" reversible computing!
Thanks for the replies!
Yeah I noticed it seems to be quite popular amongst Unity devs. So thanks for creating it whydoidoit! Why did you do it?!
I'm making a Trie to store a list of words from an english dictionary. Around 100k words. So the tree creation is very slow.
What do you think about the below options?
Read data into a Dictionary/Hashset as they have O(1) lookup and should be faster to create than a complex Trie.
Don't even read the words in. Instead, whenever you need a match, perform a Binary Search with a file stream, by pointing the stream to halfway through file, and halving back and forth to narrow search results.
Learn and integrate UnitySerializer and manually Serialize and Deserialize my Trie structure to disc as bytes.
Combine 1 and 3 by using Dictionary/Hashset with UnitySerializer.
My main concern, again, is the Deserialization/Loadtime, since I can pre-process the initial Serialization to disc.
Thanks!
EDIT: First test with HashSet proved very fast at loading/creating! Tests also seem unnoticeable, but I'm testing on a fast workstation... Next step, test on the iPad. Will post results. BTW this cut out like 100s of lines of code (Trie, Node, threading, etc)
EDIT: Test with HashSet proved quite successful! I got about a 5 second lag hit when building the HashSet from the serialized string[]. Much faster than the Trie creation! Now I'm thinkin to put that load onto another thread or just add it to the map loading. :D
