Camera click and drag pan with UI elements blocking raycast

Hi, I have a simple 2D camera panning script that moves the camera based on mouse delta movement. It works great, but I need the UI to block the panning. Here is the code:

 // This script is on the GameObject with the Camera
 public Class PanScript : Monobehaviour
 {
     private Camera camera;
     public float moveSensitivity = 0.00333f;
     private Vector3 lastMousePos = Vector3.zero;
     
     private void Awake () {
         camera = GetComponent<Camera>();
     }
     
     private void Update () 
     {
         // This adjusts for the size of the camera
         float calcMoveSens = camera.orthographicSize * moveSensitivity;
 
         if(Input.GetMouseButtonDown(0)){
             lastMousePos = Input.mousePosition;
         }
         if(Input.GetMouseButton(0))
         {
             // moves the camera by the delta mouse movement    
             transform.position -= (Input.mousePosition - lastMousePos) * calcMoveSens;
             lastMousePos = Input.mousePosition;
         }
     }
 }

I tried using EventSystems' drag handlers, but I am not really sure how to implement them as the Camera doesn't have any colliders.

I am sure that this problem has already been figured out before, or maybe I am coming at it all wrong. Anyway, thanks for the help!