Memory usage increase when voxels deleted?!

Ok, so I've never really used forums much before so please be patient with me. Anyway, I'm trying to develop a voxel/cube-based game (original right) and I am doing so using runtime/script-based mesh creation (defining vertices, triangles, etc).

However, to populate my world, I also need models, not just terrain, and so I have created a separate program for creating models and saving them as prefabs to be dragged into the world where I wish. This was working fine up until I tried building something quite a bit bigger than normal (50 x 60 x 70 block workspace) and my program started to slow down. A LOT.

It got to the point where my computer monitors suddenly blacked out, flickered, came back on long enough for me to see a Unity Out Of Memory error message, and blacked out again. Permanently.

So, upon restarting and testing a few things I have found that as blocks are added to the mesh/model, the memory usage from Unity increases (fair enough). But upon deleting blocks it also increases, to the point where I may have essentially built and cleared several models and am back with an empty mesh, but the memory usage is still 100s of MB more than when I started!

Obviously I can't keep this and must find a solution soon, so I turn to you. Please, any help/advice will be gratefully accepted! Please let me know if you require code snippets, screenshots, etc. Thank you!!