Move Objects with Collision Without Impacting Frame Rate

I have tried three different methods to move gameobjects with rigidbodies and character controllers on them.

I've tried: Transform.translate Transform.position CharacterController.SimpleMove

With the first two no matter how many objects are on the screen, the frame rate stays the same, but with the last one (really anything when dealing with character controllers) the frame rate drops into the single digits when any more than 20 objects are in the scene at once.

But no matter what, all the time the objects will simply not detect each other's collision radii. Leading to all the objects being stacked on top of each other. So how do I avoid this AND save the frame rate at the same time?