koobas.hobune.stream
Best Way to Load 10^8 int Array in Unity/C#?
Hello,
I am building an AI to play Tetris and have compiled a large heuristic function in C#. Right now, the function is saved as a text file that I load into an array when I want to manipulate it. This 10^8 int array maps a state number to a heuristic value in this way:
array[state] = heuristic
I also have a Tetris platform built in C#/Unity. I would like to be able to access my heuristic information within this platform. However, Unity takes a very long time to read the file into memory.
I have looked into using a local database to read the information, but cannot find an easy one to set up with C# (most information I have found has to do with setting up DBs for MMO characters, which is a different problem).
I need a way to quickly load the entire heuristic into memory and perform lookups while the algorithm runs. Do you have any ideas or similar experience that could help me out?
Thanks!
I would pre-process that text file into a binary file with that data. 16 or 32bit ints? Either way, it's a big file. You could use the standard Stream-related API, but I'd put it in another thread for sure. And really I would look into making probably a separate (set of) file(s) which index into that large file, then only read the parts needed at any given time. I hope you don't want this on mobile like that. For mobile, I'd access it on a server via WWW and scripts on the server.
The text files I have now are right at 1GB!
I read the entire file into an array before my game begins execution. This works fine, it just takes a few $$anonymous$$utes to load before the game begins.
It's going to. The best you can do if you want to parse that as text each time is put it in another thread or coroutine so it doesn't make you wait a few $$anonymous$$utes before you see anything.
Can you suggest another method for storing the data that will improve load time? $$anonymous$$y only thought is to store it in a local database, but I haven't been able to find an easy method for doing that in C# with Unity.
As I said, store it as an array of ints (binary). Assu$$anonymous$$g 32-bit ints, that's 4 bytes per item, and you should be able to pretty much slam it into memory. Check http://www.dotnetperls.com/binaryreader and http://www.codeproject.com/Articles/10750/Fast-Binary-File-Reading-with-C
Thank you!
I converted my text file to binary and it loads a lot faster (~25 seconds as opposed to a few $$anonymous$$utes). This will really help with debugging my AI.
