Head Look Controller with camera

I have integrated the Head Look Controller script in my character and it works like a charme. Now in stead of a GameObject I want it to look at the Main Camera which can be controlled by the user. It all works but at a certain point the camera goes all over the place. I think it has something to do with the code in the "Cursor Hit" c#.

 using UnityEngine;
 using System.Collections;
 
 public class CursorHit : MonoBehaviour {
     
     public HeadLookController headLook;
     private float offset = 1.5f;
     
     // Update is called once per frame
     void LateUpdate () {
         if (Input.GetKey(KeyCode.UpArrow))
             offset += Time.deltaTime;
         if (Input.GetKey(KeyCode.DownArrow))
             offset -= Time.deltaTime;
         
         Ray cursorRay = Camera.main.ScreenPointToRay(Input.mousePosition);
         RaycastHit hit;
         if (Physics.Raycast(cursorRay, out hit)) {
             transform.position = hit.point + offset * Vector3.up;
         }
         
         headLook.target = transform.position;
     }
 }

What I want is that the camera stays under control of the user. I've tried to alter the code but I don't know what to delete to just simplify it as the lookat target.

Thanks