How to rotate camera with offset using torque to look at target

I have a camera with a rigidbody which I move and rotate by applying forces and torques to. I'm trying to make the camera with an offset automatically look at the target / player.

The camera is moved using a standard Seek function. The offset is applied using the transform directions of the camera.

 void FollowPlayer () {
     GameObject player = GameObject.FindGameObjectWithTag("Player");

     // offset with the direction of this transform
     Vector3 offset = this.transform.TransformDirection (offsetPositionFromPlayer);
     Vector3 targetDestination = player.transform.position + offset;

     Seek (targetDestination);
 }

The torque required to make the camera look a the target is calculated like this:

 Vector3 torqueLookRotation (Transform target) {

     Vector3 targetDelta = target.position - this.transform.position;
     
     float angleDiff = Vector3.Angle (this.transform.forward, targetDelta);
     
     Vector3 cross = Vector3.Cross (this.transform.forward, targetDelta);

     Vector3 torque = cross * angleDiff;
     
     return  torque;
 }

What is the problem? The camera is constantly rotating when given an offset.

Why does it happen? The torque applied is making the camera's forward direction point at the target. The offset is moving the forward direction, making the camera never reach its target.

So, what I'm asking: Without using other game objects; what is the most elegant way of solving this problem? Have been trying to solve it for days! Any thoughts appreciated :)