How can I modify this rotation code to change how far the object rotates?

I'm using the following code to rotate an object on the left of the camera towards the camera. When the object is facing horizontally to the side of the camera, like this: |c the script will tilt the object towards the camera when triggered as intended. However, if the object starts at an angle to the camera, like this: /c it tilts away. I've tried switching the positions and rotations to local and dividing the direction and transform.right by various values, but nothing seems to work. Help? Ideally I want to reverse and reduce the direction of rotation. I can reverse the rotation by removing the negative on transform.right, but the object still rotates much too far.

             Vector3 direction = MainCamera.GetComponent<Transform>().position - transform.position;
 
             Quaternion toRotation = Quaternion.FromToRotation(-transform.right, direction);
 
             while (transform.rotation != toRotation)
             {
                 transform.rotation = Quaternion.Lerp(transform.rotation, toRotation, speed * Time.deltaTime);
                 yield return new WaitForEndOfFrame();
 
             }