RenderTexture.GrabPixels slowdown - How to fix?

I have a shader that is being applied to objects in a certain layer. Everything works fine until I add a lot. I tracked down the slowdown and found that it was the RenderTexture.GrabPixels in the Shader.SetPass. Here is what the profiler looks like.

This is what the RenderTexture.GrabPixels shows when I click on it in the profiler. Are each of these objects being handled separately? If so, is there a way to group them? (Again, only objects in a specific layer are being handled here)

Now obviously when I am in a location that the camera does not have sight of these objects, the GPU usage goes down.