Smooth Camera/Object Movement

So what I have is an orthographic camera that smoothly follows an object by "lerp-ing" to its position, however the object follows the cursor, by sending a raycast from the camera onto the screen. It all works but the object jitters while following the cursor, this is because since the camera is always moving itself, the cursor and subsequently the raycast are both always moving. I'm sure this is the cause because I ran a test with a camera that statically follows the object and it works well. I've already tried making another camera that statically follows the object for the raycast and one that "lerps". This doesn't work because the game gets confused as to which camera should be used for each task. Anyone have any ideas of how I can fix this "jittery" problem?

I'm not going to post the entire code because it's very long but here's the important part, newPos is just the position of the object, cam speed is just the speed of the camera, here we are changing the position of the camera with the "transform.position = " part, the code directly below is for the camera:

 transform.position = Vector3.Lerp(transform.position, newPos, camSpeed * Time.deltaTime);

This is what I'm using to move the object, this works perfectly, but the camera movement is making it jitter.

 var mousePos = Input.mousePosition;
         
 movePosition = Camera.main.ScreenToWorldPoint (Vector3 (mousePos.x, mousePos.y, 1));
         
 var moveDir = movePosition - transform.position;
             
 var distance = Vector3.Distance (transform.position, movePosition);
     
 rigidbody.AddForce (moveDir.normalized * speed);
 rigidbody.drag = distance +3;