Unity crashes sporadically when threading

Question

Hi! I've been working on a procedural planet generator.

Needless to say, it requires threading to run smoothly. It works well for about 5 minutes, but after a while Unity eats up about 7 GB of RAM and crashes. When built and run, it crashes with error unity virtualalloc remapping failed. I know that the Unity API isn't threadsafe, but my program only uses Vectors for threading, which should be fine?

Here are the threaded parts of the code :

 ConcurrentQueue<TerrainThreadInfo<TerrainData>> terrainThreadInfoQueue = new ConcurrentQueue<TerrainThreadInfo<TerrainData>>();
 ConcurrentDictionary<string, MeshData> storedMeshData = new ConcurrentDictionary<string, MeshData>();

 public void RequestTerrainData(Action<TerrainData> callback)
 {
     ThreadStart threadStart = delegate { TerrainDataThread(callback); };
     Thread thread = new Thread(threadStart);
     thread.IsBackground = true;
     thread.Start()
 }

 public void TerrainDataThread(Action<TerrainData> callback)
 {
     int size = root.GetSize(root);
     Vector3[] verts_holder = new Vector3[(QuadBuilder.res + 1) * (QuadBuilder.res + 1) * size];
     Vector3[] normals_holder = new Vector3[(QuadBuilder.res + 1) * (QuadBuilder.res + 1) * size];
     Vector2[] uvs_holder = new Vector2[(QuadBuilder.res + 1) * (QuadBuilder.res + 1) * size];
     Color[] colors_holder = new Color[(QuadBuilder.res + 1) * (QuadBuilder.res + 1) * size];
     int[] tris_holder = new int[(QuadBuilder.res) * (QuadBuilder.res) * 6 * size];
     TerrainData result = new TerrainData(verts_holder, normals_holder, uvs_holder, tris_holder, colors_holder, storedMeshData);
     result = GenerateMeshData(root.GetLeafNodes(root), result);
     lock(terrainThreadInfoQueue)
     {
         terrainThreadInfoQueue.Enqueue(new TerrainThreadInfo<TerrainData>(callback, result));
     }
 }

 void OnTerrainDataRecieved(TerrainData data)
 {
     mesh = new Mesh();
     mesh.SetVertices(data.GetVertices());
     mesh.SetTriangles(data.GetTriangles(), 0);
     mesh.SetNormals(data.GetNormals());
     mesh.SetUVs(0, data.GetUVs());
     mesh.SetColors(data.GetColors());
     storedMeshData = new ConcurrentDictionary<string, MeshData>(data.GetCache());
     meshFilter.sharedMesh = mesh;
     readMeshData = true;
 }

 TerrainData GenerateMeshData(QuadTreeNode[] Leafnodes, TerrainData data)
 {
     int[] t_tris = QuadBuilder.quadTemplateTriangles[15];
     for (int quad = 0; quad < Leafnodes.Length; quad++)
     {
         QuadTreeNode node = Leafnodes[quad];
         if (data.GetCache().ContainsKey(node.hashvalue))
         {
             MeshData cached;
             if (data.GetCache().TryGetValue(node.hashvalue, out cached))
             for (int i = 0; i < (QuadBuilder.res + 1) * (QuadBuilder.res + 1); i++)
             {
                 data.GetVertices()[i + quad * (QuadBuilder.res + 1) * (QuadBuilder.res + 1)] = cached.GetVerts()[i];
                 data.GetNormals()[i + quad * (QuadBuilder.res + 1) * (QuadBuilder.res + 1)] = cached.GetNormals()[i];
                 data.GetColors()[i + quad * (QuadBuilder.res + 1) * (QuadBuilder.res + 1)] = cached.GetColors()[i];
                 data.GetUVs()[i + quad * (QuadBuilder.res + 1) * (QuadBuilder.res + 1)] = cached.GetUVs()[i];
             }
         }
         else
         {
             // Values to save to cache after mesh is generated. This improves performance the next time we generate
             // data on this node
             Vector3[] c_verts = new Vector3[(QuadBuilder.res + 1) * (QuadBuilder.res + 1)];
             Vector3[] c_norms = new Vector3[(QuadBuilder.res + 1) * (QuadBuilder.res + 1)];
             Vector2[] c_uvs = new Vector2[(QuadBuilder.res + 1) * (QuadBuilder.res + 1)];
             Color[] c_colors = new Color[(QuadBuilder.res + 1) * (QuadBuilder.res + 1)];
 
             for (int x = 0, i = 0; x < (QuadBuilder.res + 1); x++)
             for (int z = 0; z < (QuadBuilder.res + 1); z++, i++)
             {
                 // Seams are solved by a skirt on every tile. Ugly, but works
                 Vector3 cornerTL = node.cornerTL;
                 Vector3 cornerTR = node.cornerTR;
                 Vector3 cornerBL = node.cornerBL;
                 Vector3 cornerBR = node.cornerBR;
                 Vector3 interpolated = Vector3.Lerp(Vector3.Lerp(cornerTL, cornerTR, (float)(x - 1) / (float)(QuadBuilder.res - 2)), Vector3.Lerp(cornerBL, cornerBR, (float)(x - 1) / (float)(QuadBuilder.res - 2)), (float)(z - 1) / (float)(QuadBuilder.res - 2));
                 interpolated = interpolated.normalized * radius;
                 Vector4 heightNormalData = GetHeightNormal(interpolated);
                 Vector3 normal = GetNormal(heightNormalData, interpolated);
                 Vector3 position = GetDisp(heightNormalData, interpolated);
                 Vector2 uv = new Vector2(z / (float)QuadBuilder.res, x / (float)QuadBuilder.res);
                 Color color = GetVertexColor(position);
 
                 if (z == 0 || z == QuadBuilder.res || x == 0 || x == QuadBuilder.res)
                 {
                     position = position.normalized * radius * 0.95f;
                 }
 
                 data.GetVertices()[i + quad * (QuadBuilder.res + 1) * (QuadBuilder.res + 1)] = position;
                 data.GetNormals()[i + quad * (QuadBuilder.res + 1) * (QuadBuilder.res + 1)] = normal;
                 data.GetUVs()[i + quad * (QuadBuilder.res + 1) * (QuadBuilder.res + 1)] = uv;
                 data.GetColors()[i + quad * (QuadBuilder.res + 1) * (QuadBuilder.res + 1)] = color;
                 c_verts[i] = position;
                 c_norms[i] = normal;
                 c_uvs[i] = uv;
                 c_colors[i] = color;
             }
 
             // Save newly generated MeshData to cache
             if (data.GetCache().Count > 5000)
                 data.GetCache().Clear();
             data.GetCache().Add(node.hashvalue, new MeshData(c_verts, c_norms, c_uvs, c_colors));
         }
         
         // Triangulation
         for (int i = 0; i < QuadBuilder.res * QuadBuilder.res * 6; i++)
         {
             data.GetTriangles()[i + quad * QuadBuilder.res * QuadBuilder.res * 6] =
                     !flipNormals
                 ?    t_tris[t_tris.Length - 1 - i] + quad * (QuadBuilder.res + 1) * (QuadBuilder.res + 1)
                 :    t_tris[i] + quad * (QuadBuilder.res + 1) * (QuadBuilder.res + 1);
         }
     }
     return data;
 }

storedMeshData contains a dictionary of previously generated chunks, so that not all chunks have to be rebuilt if not necessary. TerrainThreadInfoQueue is simply a queue of finished TerrainData that is sent to the MeshFilter for displaying. I would guess that the issue lies within GenerateMeshData, since I've seen similar implementations of threading that don't seem to have this issue.

Answer 1

Best Answer

Answer by andrew-lukasik · May 06 at 02:34 PM

Looks like a memory leak or adjacent issue. Also, your code is written in a unnecessarily wasteful manner which might have obfuscated a bug somewhere here.

No idea why but (QuadBuilder.res + 1) * (QuadBuilder.res + 1) repeats 19 times where 3 is enough.

Some programming advice by example:

// this is wasteful:

 for (int i = 0; i < (QuadBuilder.res + 1) * (QuadBuilder.res + 1); i++)
 {
     data.GetVertices()[i + quad * (QuadBuilder.res + 1) * (QuadBuilder.res + 1)] = cached.GetVerts()[i];
     data.GetNormals()[i + quad * (QuadBuilder.res + 1) * (QuadBuilder.res + 1)] = cached.GetNormals()[i];
     data.GetColors()[i + quad * (QuadBuilder.res + 1) * (QuadBuilder.res + 1)] = cached.GetColors()[i];
     data.GetUVs()[i + quad * (QuadBuilder.res + 1) * (QuadBuilder.res + 1)] = cached.GetUVs()[i];
 }

// the same results, but more frugal (less work for a cpu):

 var dv = data.GetVertices();
 var dn = data.GetNormals();
 var dc = data.GetColors();
 var du = data.GetUVs();
 
 var cv = cached.GetVerts();
 var cn = cached.GetNormals();
 var cc = cached.GetColors();
 var cu = cached.GetUVs();
 
 int numIndices = (QuadBuilder.res + 1) * (QuadBuilder.res + 1);
 for( int ic=0 ; ic<numIndices ; ic++ )
 {
     int id = ic + quad * numIndices;
     dv[id] = cv[ic];
     dn[id] = cn[ic];
     dc[id] = cc[ic];
     du[id] = cu[ic];
 }

As it happens, a frugal code (cpu-wise) is also easier to understand for humans.

I suggest you rewrite that GenerateMeshData method with this is mind.

Add comment · Show 5 · Share

Dag-Tholander · May 06 at 02:46 PM 0

Share

@andrew-lukasik Thank you for your reply! I am aware that the code needs to be cleaned up, the plan is to refactor some of the bigger methods as well :) I've managed to trace the problem to the cache. My guess is that more data is added to the dictionary than what is possible which causes the leak.

andrew-lukasik Dag-Tholander · May 06 at 02:59 PM 0

Share

Memory is a limited resource so remember to release data you no longer need. Allocating and never deallocating will lead to out of memory kind of error, sooner or later (100%). I suggest you cap the size of that storedMeshData to some specific length and remove the oldest entry once the limit is about to be reached.

Also - remember that to release mesh memory you need to call Destroy( mesh ).

Dag-Tholander andrew-lukasik · May 06 at 04:28 PM 0

Share

Thing is that it is already clamped to a size of 1000 objects, which should not be any issue. Yes, I've been looking at the new burst compiler and DOTS system. Seems great, but from what I've heard documentation is limited and the learning curve seems kind of steep. But I will look into it! Thanks for your help!

Show more comments

andrew-lukasik Dag-Tholander · May 06 at 03:05 PM 0

Share

Alternatively I suggest to switch to a job system. Because once you master BurstCompile with Unity.Mathematics sufficiently - this code will become so fast that you won't need to cache it this aggressively anymore.

Unity crashes sporadically when threading

1 Reply

Your answer

Follow this Question

Related Questions