- Home /
C# How do I read an external text file asynchronously in Unity?
Hi, I'm currently working on a visualization which required text file reading during run-time, and there will be a progress bar showing how much of the reading is done. Here's the problem; the text file is relatively large and takes roughly 20~ seconds to finish reading. During the reading Unity froze and you can't actually see the progress bar updating.
I've thought of a few ways to read the file async but I've encountered a bit of problem.
First I've tried using BackgroundWorker which will run the file-reading function in a background thread, the code goes something like this;
BackgroundWorker bgWorker = new BackgroundWorker ();
private void bgWorker_DoWork(object sender, DoWorkEventArgs e)
{
readFile ();
}
void Start () {
maxSize = new Vector3 (150, 30, 150);
curSize = maxSize / 2;
progressPercentage = 0;
renderCubes = new List<TemperatureCubes> ();
bgWorker.DoWork += new DoWorkEventHandler (bgWorker_DoWork);
bgWorker.RunWorkerAsync ();
}
readFile()
List<RowData> data;
void readFile ()
{
data = new List<RowData>();
var reader = new StreamReader (File.OpenRead (Application.dataPath + "/heatmap_output_high_res_run.csv")); //heatmap_output_all_one_cm
noOfLines = File.ReadAllLines (Application.dataPath + "/heatmap_output_high_res_run.csv").Count ();
int lineCount = 0;
while (!reader.EndOfStream)
{
RowData tempData = new RowData();
var line = reader.ReadLine ();
lineCount++;
var values = line.Split (',');
if (values.Length == 6)
{
if (int.TryParse (values [0], out tempData.timeStep)
&& int.TryParse (values [1], out tempData.level)
&& float.TryParse (values [2], NumberStyles.Float, CultureInfo.InvariantCulture, out tempData.pos.x)
&& float.TryParse (values [3], NumberStyles.Float, CultureInfo.InvariantCulture, out tempData.pos.y)
&& float.TryParse (values [4], NumberStyles.Float, CultureInfo.InvariantCulture, out tempData.pos.z)
&& float.TryParse (values [5], NumberStyles.Float, CultureInfo.InvariantCulture, out tempData.temperature)
)
{
tempData.timeStep += 1; //somedatafix
tempData.pos.x /= 100;
tempData.pos.y /= 100;
tempData.pos.z /= 100;
data.Add(tempData);
}
else
{
continue;
}
}
if(lineCount == Mathf.Floor(noOfLines*0.1f))
{
progressPercentage = 10;
}
}
}
RowData
public class RowData
{
public int timeStep;
public int level;
public Vector3 pos;
public float temperature;
public RowData()
{
timeStep = -1;
level = -1;
pos = new Vector3();
temperature = -1;
}
}
Edit: File size is roughly 150mb~, 5.7M+ lines.
However this gives an error saying:
get_dataPath can only be called from the main thread. Constructors and field initializers will be executed from the loading thread when loading a scene. Don't use this function in the constructor or field initializers, instead move initialization code to the Awake or Start function.
I'm not sure if there's any way around it, if there is please enlighten me.
Then I researched google a bit on reading text asynchronously and found out that C# has this. I tried it but turns out that Unity doesn't have Threading.Task framework(?). I'm not 100% familiar with it myself.
Basically I've tried all that I can think of but still can't figure a way around it. Is there other approach in reading text file asynchronously in Unity or fix for the methods I've tried? Thanks in advance!
Edit 2: Tried coroutine as suggested by @Bonfire Boy, by changing "void readFile()" to "IEnumerator readFile()", and in also update, if press space "StartCoroutine("readFile");"
It's notably faster, took roughly 3~ seconds to read the file (twice because of counting lines), however during that time Unity still freezes and progress on that bar can't be seen. Am I doing it wrong?
Edit 3: TL;DR Main issue: I needed to use an Unity API to read the text file;
var reader = new StreamReader (File.OpenRead (Application.dataPath + "/heatmap_output_high_res_run.csv"));
However it can only be run in the main thread, and that causes a 20~ seconds (3~ seconds using Coroutine) spike. I needed a progress bar to show the progress of the reading but during the spike Unity freezes can no progress can be seen on the bar, which defeats the purpose of the progress bar. Is there any way around it if I really do need the progress bar?
Edit 4:
I've tried moving Application.dataPath outside of readData()
string filePath = Application.dataPath + "/heatmap_output_high_res_run.csv";
and in readData,
var reader = new StreamReader (File.OpenRead (filePath));
noOfLines = File.ReadAllLines (filePath).Count ();
however this is still giving me the same error.
Ah, that's true. Coroutines plus using a WWW object to read the file, then?
Hmm, I'm not sure the main file load is atomic here. Looks like the file is being loaded twice, once just to get the number of lines and use it in a progress bar (or something, that part of the code looks like work-in-progress), and then again one line at a time using the StreamReader. Only the first of those loads is atomic.
I'd probably be looking to get the number of lines another way (eg put it at the start of the file). Or forget about tracking progress (I usually use a spinny loading indicator ins$$anonymous$$d when I don't know the size in advance).
@Bonfire Boy You are right about loading the file twice. I'm using the numbers for the progress bar that is still W.I.P.
$$anonymous$$ay I know more about the "spinny loading indicator" you mentioned? If that is a better approach I might use that ins$$anonymous$$d of a progress bar.
@Bonfire Boy Tried coroutine as you suggested, by change "void readFile()" to "IEnumerator readFile()" and in update, if press space "StartCoroutine("readFile");"
It's notably faster, took roughly 3~ seconds to read the file (twice because of counting lines), however during that time Unity still freezes and progress on that bar can't be seen updating as it still runs on the main thread.
Answer by Kiwasi · Jun 11, 2015 at 08:11 AM
Just move the call to Application.dataPath out to the main thread.
This error should go away then. Note I haven't read the rest of the code to see if it will throw any other errors.
I've tried doing this by doing this outside of readData()
string filePath = Application.dataPath + "/heatmap_output_high_res_run.csv";
and in readData,
var reader = new StreamReader (File.OpenRead (filePath));
noOfLines = File.ReadAllLines (filePath).Count ();
however this is still giving me the same error.
Answer by Bunny83 · Jun 10, 2015 at 03:41 PM
I'm pretty sure you read the error the wrong way. StreamReader should work fine in a seperate thread. What you can't use in a seperate thread are things from the UnityAPI. Since the only thing you seem to use from Unity is "Application.dataPath" i suspect that the getter is actually throwing that error. Maybe post your actual error (with stacktrace) from the console.
Btw: File.ReadAllLines reads already the whole file into a string array- You basically open and read the file two times. However you just use the Linq extension Count() to get the length and then you throw the array away.
You might want to read in the file once as a string array in the main thread and then just pass the array to your method and not using the stream reader at all. Also you did not show where you initialize your "data" variable. I guess it's a generic List? Did you specify a capacity? If not this will create a huge amount of garbage as the internal array has to be resized several times.
About how many lines do we talk about? 10k? 100? 1M? 10M? 100+M?
Furthermore what's RowData? a class or a struct? A class would be bad for performance.
A bit more details on your setup and i can suggest a better approach.
If it is Application.dataPath that's causing the issue then it could just be passed through from the main thread.
You are right, there error I'm getting is actually "get_dataPath can only be called from the main thread. Constructors and field initializers will be executed from the loading thread when loading a scene. Don't use this function in the constructor or field initializers, ins$$anonymous$$d move initialization code to the Awake or Start function. ". I'll take some time and edit the post with what you suggested.
Edit: "data" variable is a List of RowData, I've added the code in the question. The text file size is roughly 150mb~, 5.7$$anonymous$$ lines. RowData is a class.
@robbagus: The List class will start with an initial capacity of 4 items. Every time it runs out of elements in it's internal array it will create a new array with double the size and copy all elements from the old into the new array. So when you call Add 5 million times you're going to create arrays of size:
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536
131k 262k 524k 1048k 2$$anonymous$$ 4$$anonymous$$ 8$$anonymous$$
Since it's an array of a reference type the actual memory size is 4 times larger. At the end you only keep the 8$$anonymous$$ array and all others will be garbage.
If you however create the List with an initial capacity you only create one array with the correct size. Just pass the capacity to the constructor of your List. Since you read the line count before you add your elements you can set the capacity easily.
Next thing is you might want to switch your class with a struct (unless you need it to be a class). Classes have a slightly larger overhead and require the system to reserve a memory area on the heap, 5 million times... When using structs the List / array will be larger, but in the end you actually save memory and especially performance.
When using a struct i would recomment to use an array ins$$anonymous$$d of a List (unless you need to add remove items later). An array allows direct access to struct members, a List does not (only read access since access is done via indexer property).
Your answer
Follow this Question
Related Questions
Multiple Cars not working 1 Answer
Distribute terrain in zones 3 Answers
Retrieve and assign data from a .txt file? 1 Answer
Will System.IO.File work on Macs 2 Answers
How Do I Write to a Created File? 1 Answer