Dynamic colliders without Rigidbodies?

So I'm currently implementing a server authoritative networking for my game, I'm using a third party networking framework (Photon Bolt) so I can't use Rigidbodies in the game due to server sync reasons. The game is also 2D, so I don't have the Character Controller alternative, which is generally used as a Rigidbody replacement.

I did manage to get the basic movement and collisions working by script (on objects without rb or char controllers), but my thought is that if my GameObject have colliders and don't have either a Rigidbody or a CharacterController component, that means the whole static collider data is being recalculated in every frame which is slow and could lead to bugs in the near future.

Would there be a less expensive or safer way to do this, like marking just the GameObjects that will move to be recalculated every frame (aka dynamic)?