How to handle a MASSIVE amount of game objects?

My team and I are new to Unity, and have been struggling for the last several weeks with one of three problems: either high memory usage or low frame rate or frame rate spikes. We are developing a 2D game (using 3d physics because we are using character controller) that uses procedurally generated maps similar to Terraria and Starbound.

Our smallest map is 4096 blocks wide by 1024 blocks high. For a total of 4,194,304 blocks. Each block game object contains a: SpriteRenderer, box collider, BlockObj script and BlockData script.

Unity will not allow us to create this many blocks on even a small map, much less a large without exploding, so we have implemented chunks to help alleviate the issue. Chunks allow us to set the biome for that chunk, and each chunk will be X blocks wide and X blocks high. Each contain contains a block array, and a box collider (for player entered chunk triggering).

We only load the blocks contained in chunks which either: contain the player, or border a player containing chunk. We implemented a coroutine on block loading to reduce the spikes, but then our frame rate drops while the chunk blocks are loading.

We also have a garbage method which takes chunks that were loaded, and no longer border a player containing chunk and is not a player containing chunk itself. Blocks in the garbage list get destroyed in a coroutine over time.

We poll a special class every 1 second to update player containing chunks, player bordering chunks, and garbage chunks. The system is optimized to not reinstantiate or create duplicates.

With the current system we have excellent memory usage, typcially below 150 MB for a small map. But depending on what settings we tweak whether it be the rate at which we update garbage, player chunks, player bordering chunks (every second), or the rate at which blocks are created/destroyed within their coroutines (0.1f), it just doesn't matter. We either have unacceptable frame rate spikes or frame rate drops over time.

We have tried several other methods in the past as well such as creating faces and turning the blocks into a chunk mesh. This resulted in crazy high memory usage (900+ MB for a small map) but had excellently stable frame rate. We also looked into batching for our current setup and were disappointed to see that the batching is only useful for meshes (which we are obviously no longer using).

I am beginning to think maybe we are doing something wrong. We are new to Unity after all. We need to manage between 4,194,304 and 16,777,216 blocks at a time, in addition to other game objects such as player, flora & fauna, background tiles (for creating that lovely wooden background wall in your house i.e.), environment backgrounds (majestic mountain views!), enemies, NPCs, projectiles, dropped objects, chests, etc. What would be the best way to accomplish this?