Possible to Give Rigid bodies collision bounding boxes?

So when I do a cast (ray/sphere/capsule) on a bunch of colliders (3000+) in a scene, it runs really efficiently, IF the colliders bounding boxes are outside of the area I am casting. However if we attach a kinematic rigid body to these colliders its as if the bounding boxes are taken out and the cast checks against EVERY collider in the entire scene. Is there a way to restrict the area that these casts check even if the colliders have a rigid body? (maybe even something like occlusion with physics?)

--I assume that it is removing the bounding box filtering before it does a collision check, and hence why it becomes astronomically less efficient. I understand its probably because a rigid body has the idea of that the object will be moving (possibly very fast) and unity wants to ensure that the cast is accurate and doesn't miss the moving rigid bodies, but in my game nothing is moving that fast. So do I have to remove rigid bodies completely from my game assuming I NEED that many colliders, or is there a more elegant solution?

Thanks!