Moving a camera with AddFoce, or setting velocity, need help

This seems like the kind of thing I should be able to quickly find a reference or tutorial for, but the keywords just don't seem to lead me to anything.

So, here is my workflow:

Character Control Script

^A method called from LateUpdate to control the camera

^This method basically takes the code from "Mouse orbit improved" to orbit an empty gameobject around the player, with the mouse

^This method then uses physics to push the camera towards this orbited position.

 cameraRigidbody.velocity = ((direction) * (Mathf.Abs (Vector3.Distance ((the world position of my orbited object), (the current world position of the camera)))*200f) * (Time.smoothDeltaTime));

This was working very well but it's too shaky and inconsistent to stick with.

Please someone, what is the ideal way to move a camera with physics, PREFERABLY keeping true to the current order of operations (in that, we use one object for "where it should be" and use physics to move it towards that object")