Raycasts, Collisions, and Triggers outside of Camera frustum NOT WORKING

Okay I have been fighting with this for awhile now and I cant find anything to help on here. I saw a few people asked and one guy said yes physics are still calculated out of the frustum, which is true because my character controllers still work outside the camera but not the events I need.

Anyway I have a fps multiplayer game supporting both dedicated and listen servers all authoritative. If the player is also acting as a server and his camera is not facing the other players Raycasting on their colliders, collision events and trigger events will not go through reliably (occasionally an event will work) after they die and their ragdoll colliders and rigidbodys are activated and then deactivated. Their damage colliders which work at first will work again if the server glances at them after the ragdolls have been deactivated. The damage colliders remain on along with their rigidbodys the entire time. They are set to only collide with what I need them to in the physics layers.

I'm guessing when the servers camera glances at them it does some type of refresh on the colliders and rigidbodys? If I'm in the editor and the scene view(but not the players camera) is looking at them it will work too. If the dedicated servers camera is not oriented in a way that the players are always in the frustum it will not work.

Notes for common suggestions:

Their damage colliders all have kinematic rigidbodys on them. I am raycasting specifically at their layer. Two kinematic rigidbodys never need to collide, occasionally they need to collide with a kinematic trigger or a nonkinematic rigidbody. Yes I have tried continuous and continuous dynamic properties for the rigidbodys these had no effect. The timing is already at 0.1666 when lowered even more it still had no effect. Everything works correctly, however, if the rigidbodys are within the servers frustum. I can be sure they are being replicated properly when the server is not facing them because the server is authoritative and both clients have the same positions for each other when not faced by the camera etc. Another big problem is a camera MUST be active or no raycasts triggers etc will work at all!? This is a shame for the dedicated server.

My only solution is to have a second camera active that oversees the entire map, then activate my players camera which overrides the other camera while still keeping it active. Under this scenario all events work correctly but it severely effects the fps of the server player. I tried putting only the layers I need to collide in the layermask of the overseeing camera but it seems its an all or nothing deal for it to work properly.

I am very surprised at this behavior since others seem to suggest that yes the physics should still work outside the frustum and that I could find really no one else who had trouble with it. I can only hope that I am ignorant and there is a common solution. Thank you so much for taking the time to read this and help me! I'd love to hear your thoughts.