An alternative to RotateAround for collisions

I've got two player controlled objects that face each other at all times (named Red and Blue). Both one has a child chase camera. I want them to move towards and away the enemy, and rotate around it smoothly, when the standard 4 directions are pressed.

To move towards and away from each other, I use

 rigidbody.velocity = transform.forward * Movespeed
 

This works fine, for both the player and the child cameras. Smooth as butter.

To rotate, I tried to use RotateAround. It utterly broke the collisions, and the thing could just jump through objects because I wasn't affecting Velocity or Force, I was just translating it. So now I'm using:

 rigidbody.velocity = transform.right * Movespeed

This kind of works, since the LookAt updates every frame. But the fact that transform.right isn't the same as RotateAround means that the camera jumps and shakes every frame of movement because it has to adjust, and the player slowly moves away from the opponent when moving.

Anyone have an alternative solution? Having the camera not be a child and simply match the player's movement doesn't help, and it also doesn't rotate along with the player when I do that. FixedUpdate, DeltaTime, etc, nothing seems to help.