how to stop mouse from teleporting or detect better, so that i can detect if a swipe of mouse was over object

Hello,

I am creating a golf game where player have to swipe/flick over the golf ball in order to hit the ball, the start position and end position of the flick can be anywhere but the flick/swipe has to go over the ball in its duration

this is the code i have written till now

    void Update() {
             if (Input.GetMouseButtonUp (0) && didPassOver) {
                 didPassOver = false;
                 var endPos = Input.mousePosition;
                 endPos.z = transform.position.z - Camera.main.transform.position.z;
                 endPos = Camera.main.ScreenToWorldPoint(endPos);
                
                 var force = endPos - startPos;
                 force.x = -force.x;
                 force.z = force.y;
                 force.y = 0;
                 force /= (Time.time - startTime);          
                 rb.AddForce(force * factor);
             }
      
             if (Input.GetMouseButtonDown (0)) {
                 startTime = Time.time;
                 startPos = Input.mousePosition;
                 startPos.z = transform.position.z - Camera.main.transform.position.z;
                 startPos = Camera.main.ScreenToWorldPoint(startPos);
             }
         }
      
         void OnMouseOver() {
             if (!didPassOver) {
                 didPassOver = true;
             }
         }

Mostly this is working okay, I just have one problem that if you move mouse fast ( very easy to happen ), then the OnMouseOver doesnt gets called and it doesnt considers it as flick over the object,

any suggestion on to sort this out?

this game is targeted for mobiles mostly, but I am coding keeping in mind that web/pc might be used as well

I am open to completely different solutions as well

Thanks in advance