Is it possible to get the physics engine to ignore calculating a rigidbody?

Since you cannot disable rigidbodies (and I don't want to instantiate them, so they have to be there), is there a way to make the physics engine ignore the rigidbody to increase performance (the Physics.Simulate is a real performance hit)? I tried doing RigidbodyConstraints.FreezeAll until I needed it but that doesn't seem to help. Any ideas?