Problems with Out of Memmory (GC)

I have some problems with getting enough memory to declare some very very very large objects. its a Color[1024x1024x32] to load into a 3DTexture. Unfortunately 3DTexture does not support Color32.

Anyways, When I doing my scene terrain generation (seperate scene actualy) I used up to about 500megs (the data is then saved into a file)

Then when Load the scene that builds the 3DTexture and loads the data. It will run out of memory (throwing the exception). The GC says the 500megs is still declared.

However, if I restart unity (or the program) after running the generation. It Runs with no problems.

I've run the generator multiple times, and I doesn't show any leaks, (Never goes beyond 500, always fluctuates around it)

Also, the generation does happen in a separate thread; Any Recommendations?