- Home /
Why is reading some text files with JS so darn slow?! (title changed to reflect answers)
Here's the relevant code:
var textAsset : TextAsset;
var lineCounter = 0;
function Start() {
readHugeFile();
}
function readHugeFile() {
var startTimer = Time.realtimeSinceStartup;
if (textAsset == null) return;
reader = new StringReader(textAsset.text);
line = reader.ReadLine();
while (line != null)
{
lineCounter++;
line = reader.ReadLine();
}
var timeExpired = Time.realtimeSinceStartup - startTimer;
print("Time Expired:");
print(timeExpired);
print("Number of Lines:");
print(lineCounter);
}
Ok, by points.
I'm not interested in alternatives to reading these files. Alternative/smarter code would be super welcome!
This code is to darn SLOW when the number of lines grows.
This thing scales weirdly. The length of time taken isn't linear with the number of lines in the file.
Some random sampling of text files of different sizes.
(Edits after modification suggested)
Number of Lines Lines Read / second | *Lines Read / second
57,012 251 518,000 54,818 245 550,000 41,239 359 412,000 38,138 385 545,000 32,305 447 538,000 25,908 551 518,000 21,098 681 527,000 17,668 803 589,000 16,038 891 535,000 13,118 1161 656,000 11,616 1263 581,000 2,293 6551 573,000
WTH?
Thanks for any answers, pointers, or directions for better code. Am I doing something silly here?
*Notes The large variance in the number in the far right column are a result of variances in actual line lengths in the files, rounding error, and poor use of significant digits by myself.
With the largest file tested here, the suggested change of code represents a 2000X speed increase!
Never worked with StringReader. How long does it take to execute just the line new StringReader(textAsset.text);
? I guess the StringReader doesn't work very efficient on the input data internally. Looks really strange.
Just found this article: http://lists.ximian.com/pipermail/mono-bugs/2009-November/095088.html
Thanks Bunny! Indeed, your link explains the strange scaling. Super helpful. I hate just going "wth? I don't get it."
Answer by Eric5h5 · Feb 28, 2011 at 05:32 PM
How about
if (textAsset == null) return;
var lines = textAsset.text.Split("\n"[0]);
var lineCounter = lines.Length;
Might have to be "\r" if that's how lines are delineated.
Well "Slap my hind with a melon rind"!
Check this out! Editing table in post with new data.
Eric, this seems like a whopper of a small easily-made error. Should this be brought to anyone's attention? This teensy code-change just saved a project that I was about to give up on.
@rule62: Well, it depends on what you're doing. Using StringReader with ReadLine may be correct for certain cases. I wasn't sure if my suggestion would be appropriate for what you're doing (since I don't know what that actually is), which is why I phrased my answer as a question. :) Glad it worked for you though.
After reading Bunny83's link, it sounds like a bug report would be in order. Seems like that should have been fixed given the version of $$anonymous$$ono Unity uses?