Isometric Prespective camera drag/drop

The main camera of my game is set up as a perspective camera with rotation x:36,y:-34,z:0. It gives me a pretty nice "isometric" view of the game. This looks like this:

I want to make the camera "draggable". Not like in a RTS that the camera moves when the mous is getting out of the fiel of view, more kind of a camera drag in thouse mobile building games. I set up all of this woth an ortographic camera, which is no problem, but I'm really struggling with the perspective camera view. To move the camera I gave the ground (this visible grid) a box collider to measure the distance from mousbuttondown to the drag mposition of the mouse. It works good like this but it is insanely fast and not at all accurate. Furtheermore it drags forever and does not stop when it reaches the desired destination. My code currently looks like this:

     Vector3 startpos;
 
     void OnMouseDown() {
 
         Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
         Plane plane = new Plane(Vector3.up, Vector3.zero); 
         float distance = 0f;
         Vector3 point = ray.GetPoint(distance);
         startpos = point;
 
     }
 
 
     void OnMouseDrag()
     {
         Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
 
         Plane plane = new Plane(Vector3.up, Vector3.zero); 
         float distance = 0f;
 
         Vector3 point = ray.GetPoint(distance);
 
         Vector3 diff = startpos - point;
 
         Camera.main.transform.position = Camera.main.transform.position - diff;
 
       
     }

I tried to update the start position when finish moving like this:

         Camera.main.transform.position = Camera.main.transform.position - diff;
 

But it will just zoom in forever... Can anyone giving me an advice what I am doing wrong or how to do this?