Help changing plane.Raycast dragging script

I have a script which was initially written to drag the Player around the screen in the X-Y plane. It basically creates a Plane in front of the camera (normals facing away) with the Player on other side of it and records the point of intersection of a plane.Raycast into the plane. It then converts this local intersection position to World space and moves the Player there.

However, what I would like to do now is to only move the Player to the mouse position when he is clicked on and dragged there, rather than just clicking the mouse somewhere on the screen/plane. I have tried to include a further Physics.Raycast if- that checks if the Player collider was hit and then moves him to the position. The problem is that whilst dragging the Player, if the mouse moves off of him then he will stop moving. This makes sense as the ray knows nothing about holding a mouse button down.

Also, one of the reasons I used a plane.Raycast is so that I could create screen boundaries. These can be seen in the form of offsets on the Mathf.clamp functions near the bottom.

Any suggestions on how I could do this would be great.

The script is below.

 void FixedUpdate()
     {
         if (movementEnabled)
         {
             // Variable to represent the camera
             Camera myCamera = Camera.main;
 
             // Create a plane with normal in the opposite direction to where the Main Camera is facing,          and with a 
             // fixed position which is set distance from the camera.
             Plane plane = new Plane(-myCamera.transform.forward, myCamera.transform.position + myCamera.transform.forward * DesiredPlayerDistanceFromCam);
 
             // Create a ray to hold the information when the mouse is clicked on the screen. 
             // Returns a ray going from camera through a screen point.
             // Resulting ray is in world space, starting on the near plane of the camera and going through position's
             // (x,y) pixel coordinates on the screen (position.z is ignored)
             Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
 
             float dist = DesiredPlayerDistanceFromCam;
 
             // If mouse LMB is pressed (or held down)..
             if (Input.GetButton("Fire1"))
             {
                 // If the ray intersects the plane..
                 if (plane.Raycast(ray, out dist))
                 {
                     // Find the position on the plane and store in variable pos
                     Vector3 pos = ray.GetPoint(dist);
 
                     RaycastHit hit;
 
                     if (Physics.Raycast(ray, out hit, 100))
                     {
                         if (hit.collider.tag == "Player_Main_Collider")
                         {
                             PlayerTouched = true;
 
                             pos2 = new Vector3(pos.x, pos.y - mouseYoffset, pos.z);
 
                             // Move the Player to the position that was clicked buy the user on the plane
                             Player.position = Vector3.Lerp(Player.position, pos2, Time.deltaTime * Smooth);
 
                             // Find the Player's Local position relative to the camera.
                             // This operation is unaffected by scale.
                             Vector3 localPos = myCamera.transform.InverseTransformPoint(Player.position);
 
                             //              Debug.Log("localPos = " + localPos);
 
                             // Find the World space position of the left-bottom boundary of the viewport
                             leftBottom = myCamera.ViewportToWorldPoint(new Vector3(0, 0, DesiredPlayerDistanceFromCam));
                             //                Debug.Log("leftBottom = " + leftBottom);
 
                             // Find the World space position of the right-top of the viewport
                             rightTop = myCamera.ViewportToWorldPoint(new Vector3(1, 1, DesiredPlayerDistanceFromCam));
                             //                Debug.Log("rightTop = " + rightTop);
 
                             // Get the Local position of the leftBottom point in World Space
                             leftBottom = myCamera.transform.InverseTransformPoint(leftBottom);
 
                             // Get the Local position of the rightTop point in World Space
                             rightTop = myCamera.transform.InverseTransformPoint(rightTop);
                             //        Debug.Log("rightTop = " + rightTop);
 
 
                             //Player.position = pos2;
 
                             Vector3 pos2Local = myCamera.transform.InverseTransformPoint(pos2);
 
                             // Define the bounds to clamp for the X and Y axes
                             float x = Mathf.Clamp(localPos.x, leftBottom.x + leftRightOffset, rightTop.x - leftRightOffset);
                             float y = Mathf.Clamp(localPos.y, leftBottom.y + bottomOffset, rightTop.y - topOffset);
 
                             // Move the Player to coordinates defined by the Mathf.Clamp functions, converting them
                             // back into World Space coordinates first
                             //                Player.position = myCamera.transform.TransformPoint(new Vector3(x, y, localPos.z));
 
 
                             Player.position = myCamera.transform.TransformPoint(new Vector3(x, y, localPos.z));
                         }
                         else
                             PlayerTouched = false;
                     }
                         
                 } 
 
             } 
 
         }
 
     }