Why Does My Collision Detection Become Inconsistent?

I have a scene where I am creating level geometry by bumping a bunch of colliders into each other to create rock formations. It's probably weird and inefficient but I like it. The issue is that at a certain point it seems that the collision detection stops happening and the rocks begin to clip through each other. It's visible when this happens because the formations change, as you can see in the attached screenshot. The formation is created over time at runtime by following a wandering object and spawning bricks. The bricks collide and land in interesting craggy shapes. In the image attached you can see that at a point a transition occurs where the formation changes and smooths out because the bricks are clipping through each other and not being re-positioned by the collision physics.

This also seems to coincide with the bullet collision physics becoming inconsistent. I am not seeing a big performance hit at this point, everything continues to run at a good frame rate, up to 77 FPS. I notice that after this level geometry problem begins to occur I have situations where I shoot at my enemies and bullets pass through them 9/10 times although hitting occasionally. It's very weird and confusing. Has anyone seen issues like this where collision initially works and then starts breaking?

To add some additional context:

Upon further testing I've realized that the collision is breaking very inconsistently. For example once it begins breaking I can fly against a rock in scene and sporadically pass through it or not. This seems to happen after I spawn the third or fourth wave of enemies. Is it possible that instantiating a bunch of new game objects at once is causing this to break?

For those who are interested there is a webplayer here which: http://www.mirrorfishmedia.com/v1/pollen/pollen_v033/pollen_v033.html

Thanks for taking the time to check this out, if anything is unclear I can elaborate.

best, M.