Wayback Machinekoobas.hobune.stream
May JUN Jul
Previous capture 13 Next capture
2021 2022 2023
1 capture
13 Jun 22 - 13 Jun 22
sparklines
Close Help
  • Products
  • Solutions
  • Made with Unity
  • Learning
  • Support & Services
  • Community
  • Asset Store
  • Get Unity

UNITY ACCOUNT

You need a Unity Account to shop in the Online and Asset Stores, participate in the Unity Community and manage your license portfolio. Login Create account
  • Blog
  • Forums
  • Answers
  • Evangelists
  • User Groups
  • Beta Program
  • Advisory Panel

Navigation

  • Home
  • Products
  • Solutions
  • Made with Unity
  • Learning
  • Support & Services
  • Community
    • Blog
    • Forums
    • Answers
    • Evangelists
    • User Groups
    • Beta Program
    • Advisory Panel

Unity account

You need a Unity Account to shop in the Online and Asset Stores, participate in the Unity Community and manage your license portfolio. Login Create account

Language

  • Chinese
  • Spanish
  • Japanese
  • Korean
  • Portuguese
  • Ask a question
  • Spaces
    • Default
    • Help Room
    • META
    • Moderators
    • Topics
    • Questions
    • Users
    • Badges
  • Home /
avatar image
1
Question by robbagus · Jun 10, 2015 at 03:22 PM · c#fileasyncthreadingreading

C# How do I read an external text file asynchronously in Unity?

Hi, I'm currently working on a visualization which required text file reading during run-time, and there will be a progress bar showing how much of the reading is done. Here's the problem; the text file is relatively large and takes roughly 20~ seconds to finish reading. During the reading Unity froze and you can't actually see the progress bar updating.

I've thought of a few ways to read the file async but I've encountered a bit of problem.

First I've tried using BackgroundWorker which will run the file-reading function in a background thread, the code goes something like this;

 BackgroundWorker bgWorker = new BackgroundWorker ();
     
         private void bgWorker_DoWork(object sender, DoWorkEventArgs e)
         {
             readFile ();
         }

 void Start () {
 
         maxSize = new Vector3 (150, 30, 150);
         curSize = maxSize / 2;
         progressPercentage = 0;
 
         renderCubes = new List<TemperatureCubes> ();
 
 
         bgWorker.DoWork += new DoWorkEventHandler (bgWorker_DoWork);
         bgWorker.RunWorkerAsync ();
     }

readFile()

 List<RowData> data;
 
     void readFile ()
         {
             data = new List<RowData>();
     
             var reader = new StreamReader (File.OpenRead (Application.dataPath + "/heatmap_output_high_res_run.csv")); //heatmap_output_all_one_cm
             noOfLines = File.ReadAllLines (Application.dataPath + "/heatmap_output_high_res_run.csv").Count ();
             int lineCount = 0;
     
             while (!reader.EndOfStream) 
             {
     
                 RowData tempData = new RowData();
     
                 var line = reader.ReadLine ();
                 lineCount++;
                 var values = line.Split (',');
     
                 if (values.Length == 6) 
                 {
                     if (int.TryParse (values [0], out tempData.timeStep)
                         && int.TryParse (values [1], out tempData.level)
                         && float.TryParse (values [2], NumberStyles.Float, CultureInfo.InvariantCulture, out tempData.pos.x)
                         && float.TryParse (values [3], NumberStyles.Float, CultureInfo.InvariantCulture, out tempData.pos.y)
                         && float.TryParse (values [4], NumberStyles.Float, CultureInfo.InvariantCulture, out tempData.pos.z)
                         && float.TryParse (values [5], NumberStyles.Float, CultureInfo.InvariantCulture, out tempData.temperature)
                         )
                     {
                         tempData.timeStep += 1; //somedatafix
     
                         tempData.pos.x /= 100;
                         tempData.pos.y /= 100;
                         tempData.pos.z /= 100;
     
                         data.Add(tempData);
     
                     } 
                     else 
                     {
                         continue;    
                     }
                 }
     
                 if(lineCount == Mathf.Floor(noOfLines*0.1f))
                 {
                     progressPercentage = 10;
                 }
             }
     
             
         }

RowData

 public class RowData
     {
         public int timeStep;
         public int level;
         public Vector3 pos;
         public float temperature;
 
         public RowData()
         {
             timeStep = -1;
             level = -1;
             pos =  new Vector3();
             temperature = -1;
         }
     }

Edit: File size is roughly 150mb~, 5.7M+ lines.

However this gives an error saying:

get_dataPath can only be called from the main thread. Constructors and field initializers will be executed from the loading thread when loading a scene. Don't use this function in the constructor or field initializers, instead move initialization code to the Awake or Start function.

I'm not sure if there's any way around it, if there is please enlighten me.

Then I researched google a bit on reading text asynchronously and found out that C# has this. I tried it but turns out that Unity doesn't have Threading.Task framework(?). I'm not 100% familiar with it myself.

Basically I've tried all that I can think of but still can't figure a way around it. Is there other approach in reading text file asynchronously in Unity or fix for the methods I've tried? Thanks in advance!

Edit 2: Tried coroutine as suggested by @Bonfire Boy, by changing "void readFile()" to "IEnumerator readFile()", and in also update, if press space "StartCoroutine("readFile");"

It's notably faster, took roughly 3~ seconds to read the file (twice because of counting lines), however during that time Unity still freezes and progress on that bar can't be seen. Am I doing it wrong?

Edit 3: TL;DR Main issue: I needed to use an Unity API to read the text file;

 var reader = new StreamReader (File.OpenRead (Application.dataPath + "/heatmap_output_high_res_run.csv"));

However it can only be run in the main thread, and that causes a 20~ seconds (3~ seconds using Coroutine) spike. I needed a progress bar to show the progress of the reading but during the spike Unity freezes can no progress can be seen on the bar, which defeats the purpose of the progress bar. Is there any way around it if I really do need the progress bar?

Edit 4:

I've tried moving Application.dataPath outside of readData()

 string filePath = Application.dataPath + "/heatmap_output_high_res_run.csv";

and in readData,

 var reader = new StreamReader (File.OpenRead (filePath)); 
 noOfLines = File.ReadAllLines (filePath).Count ();

however this is still giving me the same error.

Comment
Add comment · Show 6
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Bonfire-Boy · Jun 10, 2015 at 03:25 PM 0
Share

Have you tried using a coroutine?

avatar image Bonfire-Boy · Jun 10, 2015 at 03:33 PM 0
Share

Ah, that's true. Coroutines plus using a WWW object to read the file, then?

avatar image Bonfire-Boy · Jun 10, 2015 at 03:45 PM 0
Share

Hmm, I'm not sure the main file load is atomic here. Looks like the file is being loaded twice, once just to get the number of lines and use it in a progress bar (or something, that part of the code looks like work-in-progress), and then again one line at a time using the StreamReader. Only the first of those loads is atomic.

I'd probably be looking to get the number of lines another way (eg put it at the start of the file). Or forget about tracking progress (I usually use a spinny loading indicator ins$$anonymous$$d when I don't know the size in advance).

avatar image robbagus · Jun 11, 2015 at 01:33 AM 0
Share

@Bonfire Boy You are right about loading the file twice. I'm using the numbers for the progress bar that is still W.I.P.

$$anonymous$$ay I know more about the "spinny loading indicator" you mentioned? If that is a better approach I might use that ins$$anonymous$$d of a progress bar.

avatar image robbagus · Jun 11, 2015 at 02:36 AM 0
Share

@Bonfire Boy Tried coroutine as you suggested, by change "void readFile()" to "IEnumerator readFile()" and in update, if press space "StartCoroutine("readFile");"

It's notably faster, took roughly 3~ seconds to read the file (twice because of counting lines), however during that time Unity still freezes and progress on that bar can't be seen updating as it still runs on the main thread.

Show more comments

2 Replies

· Add your reply
  • Sort: 
avatar image
1

Answer by Kiwasi · Jun 11, 2015 at 08:11 AM

Just move the call to Application.dataPath out to the main thread.

This error should go away then. Note I haven't read the rest of the code to see if it will throw any other errors.

Comment
Add comment · Show 1 · Share
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image robbagus · Jun 11, 2015 at 08:40 AM 0
Share

I've tried doing this by doing this outside of readData()

 string filePath = Application.dataPath + "/heatmap_output_high_res_run.csv";

and in readData,

 var reader = new StreamReader (File.OpenRead (filePath)); 
 noOfLines = File.ReadAllLines (filePath).Count ();

however this is still giving me the same error.

avatar image
0

Answer by Bunny83 · Jun 10, 2015 at 03:41 PM

I'm pretty sure you read the error the wrong way. StreamReader should work fine in a seperate thread. What you can't use in a seperate thread are things from the UnityAPI. Since the only thing you seem to use from Unity is "Application.dataPath" i suspect that the getter is actually throwing that error. Maybe post your actual error (with stacktrace) from the console.

Btw: File.ReadAllLines reads already the whole file into a string array- You basically open and read the file two times. However you just use the Linq extension Count() to get the length and then you throw the array away.

You might want to read in the file once as a string array in the main thread and then just pass the array to your method and not using the stream reader at all. Also you did not show where you initialize your "data" variable. I guess it's a generic List? Did you specify a capacity? If not this will create a huge amount of garbage as the internal array has to be resized several times.

About how many lines do we talk about? 10k? 100? 1M? 10M? 100+M?

Furthermore what's RowData? a class or a struct? A class would be bad for performance.

A bit more details on your setup and i can suggest a better approach.

Comment
Add comment · Show 3 · Share
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Bonfire-Boy · Jun 10, 2015 at 03:54 PM 0
Share

If it is Application.dataPath that's causing the issue then it could just be passed through from the main thread.

avatar image robbagus · Jun 11, 2015 at 01:10 AM 0
Share

You are right, there error I'm getting is actually "get_dataPath can only be called from the main thread. Constructors and field initializers will be executed from the loading thread when loading a scene. Don't use this function in the constructor or field initializers, ins$$anonymous$$d move initialization code to the Awake or Start function. ". I'll take some time and edit the post with what you suggested.

Edit: "data" variable is a List of RowData, I've added the code in the question. The text file size is roughly 150mb~, 5.7$$anonymous$$ lines. RowData is a class.

avatar image Bunny83 · Jun 11, 2015 at 05:06 AM 0
Share

@robbagus: The List class will start with an initial capacity of 4 items. Every time it runs out of elements in it's internal array it will create a new array with double the size and copy all elements from the old into the new array. So when you call Add 5 million times you're going to create arrays of size:

 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536
 131k 262k 524k 1048k 2$$anonymous$$ 4$$anonymous$$ 8$$anonymous$$

Since it's an array of a reference type the actual memory size is 4 times larger. At the end you only keep the 8$$anonymous$$ array and all others will be garbage.

If you however create the List with an initial capacity you only create one array with the correct size. Just pass the capacity to the constructor of your List. Since you read the line count before you add your elements you can set the capacity easily.

Next thing is you might want to switch your class with a struct (unless you need it to be a class). Classes have a slightly larger overhead and require the system to reserve a memory area on the heap, 5 million times... When using structs the List / array will be larger, but in the end you actually save memory and especially performance.

When using a struct i would recomment to use an array ins$$anonymous$$d of a List (unless you need to add remove items later). An array allows direct access to struct members, a List does not (only read access since access is done via indexer property).

Your answer

Hint: You can notify a user about this post by typing @username

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this Question

Answers Answers and Comments

6 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

Multiple Cars not working 1 Answer

Distribute terrain in zones 3 Answers

Retrieve and assign data from a .txt file? 1 Answer

Will System.IO.File work on Macs 2 Answers

How Do I Write to a Created File? 1 Answer


Enterprise
Social Q&A

Social
Subscribe on YouTube social-youtube Follow on LinkedIn social-linkedin Follow on Twitter social-twitter Follow on Facebook social-facebook Follow on Instagram social-instagram

Footer

  • Purchase
    • Products
    • Subscription
    • Asset Store
    • Unity Gear
    • Resellers
  • Education
    • Students
    • Educators
    • Certification
    • Learn
    • Center of Excellence
  • Download
    • Unity
    • Beta Program
  • Unity Labs
    • Labs
    • Publications
  • Resources
    • Learn platform
    • Community
    • Documentation
    • Unity QA
    • FAQ
    • Services Status
    • Connect
  • About Unity
    • About Us
    • Blog
    • Events
    • Careers
    • Contact
    • Press
    • Partners
    • Affiliates
    • Security
Copyright © 2020 Unity Technologies
  • Legal
  • Privacy Policy
  • Cookies
  • Do Not Sell My Personal Information
  • Cookies Settings
"Unity", Unity logos, and other Unity trademarks are trademarks or registered trademarks of Unity Technologies or its affiliates in the U.S. and elsewhere (more info here). Other names or brands are trademarks of their respective owners.
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Default
  • Help Room
  • META
  • Moderators
  • Explore
  • Topics
  • Questions
  • Users
  • Badges