Drag rigidbody with camera at angle?

When my camera's z axis is parallel with the rigidbody, everything works fine. The rigidbody doesn't move on the z (its a side scroller type environment, no z movement). However, once I rotate my camera to be angled at 23 degrees, my rigidbody consequently moves along the z axis. I figure this is because the view plane when using ScreenToWorldPoint isnt lined up with the rigidbody, so it causes it to move. I'm trying to drag a rigidbody with the mouse while the camera is at an angle, but lock the rigidbody's z axis. Here is a little drawing I did in photoshop to help you visualize what I'm thinking:

Here is the code I currently have:

     public Vector3 target;
 
     public Vector3 worldPoint;
     
     public Vector3 screenToWorldObj;
     
     public float smooth;
 
     
     void OnMouseDrag() {
 
         rigidbody.useGravity = false;
         
         Vector3 screenToWorldObj = Camera.main.WorldToScreenPoint(rigidbody.position);
         
         worldPoint = Camera.main.ScreenToWorldPoint(new Vector3(Input.mousePosition.x, Input.mousePosition.y, screenToWorldObj.z));
         
         rigidbody.position = Vector3.Lerp (rigidbody.position, worldPoint, smooth);
 
     }
     
     void OnMouseUp() {
         
         rigidbody.useGravity = true;
         
     }