Rigidbody interpolation problem

I am working on a game where the player character is moved using rigidbody physics and having trouble getting the movement to appear smooth when viewed through the ingame camera.

The rigidbody is moved using AddForce inside of FixedUpdate and all camera movement happens in LateUpdate.

Because of the difference between the camera and physics update rate causing the rigidbody movement to appear laggy, I have enabled interpolation on the rigidbody, but that only makes the lag far worse and also causes the rigidbody to move at wildly inconsistent speeds.

What could I be doing wrong here?