addforce + persp camera movement weirdness

I'm using mouse position to control the movement of an object on screen, basically wherever you click pushing the object in the opposite direction and how long you hold the mouse down for the speed.

Mostly it's fine, but every few clicks or seconds it comes back with the wrong direction pulling the object toward the mouse instead of pushing it away.

Using an ortho camera instead of persp seems to work fine, so it's something to do with that but I can't figure out how to fix it. And turning off the oscillation doesn't help.

Here's my code:

    function Start()    {
     
         particleEmitter.emit = false;
         yield WaitForSeconds(5);
         particleEmitter.emit = true;
     }
     
     
     function Update () {
         
         if(Input.GetButtonDown("Fire1"))    {
         
             clickStart = Time.time;
             mousePos = Camera.main.ScreenToWorldPoint(Vector3(Input.mousePosition.x, Input.mousePosition.y, Camera.main.transform.position.z)); 
             currentPos = Vector3(transform.position.x, transform.position.y, transform.position.z);
             
             moveDir = mousePos - currentPos;        // persp camera, change for orth
             moveDir.z = 0;                                //don't move the object on the z axis
             moveDir.Normalize();
         }
     
         if(Input.GetButtonUp("Fire1"))    {
     
             clickEnd = Time.time;
             
             //speedCalc = Vector3.Distance(mousePos, currentPos);        //distance pointer from seed to set the speed
             speedCalc = clickEnd - clickStart;                        //time mouse button held to set the speed
             
             vel = Mathf.Clamp((speedCalc / moveSpd), minVel, maxVel);
             rigidbody.AddForce(moveDir * vel, ForceMode.Impulse);
         }
         
         oscillatePos = Vector3(transform.position.x, transform.position.y, transform.position.z);
         transform.position.y = oscillatePos.y + Mathf.Sin(Time.time * 1.5) * oscillateDist;
         transform.position.x = oscillatePos.x + Mathf.Sin(Time.time * 1) * oscillateDist;
         
     }