ScreenPointToRay(Input.mousePosition) - taking into account the first touch

I have a standard script where is dragging object:

     if (Input.GetMouseButton(0))
 { 
         var ray = Camera.main.ScreenPointToRay(Input.mousePosition);  
         if (!dragObj )
 {
             
             if (Physics.Raycast(ray, hit) && hit.rigidbody)
 {
                 dragObj = hit.transform;
                 length = hit.distance; 
                 
             }
         } 
         else {  
 
         // if some object was picked...
             // calc velocity necessary to follow the mouse pointer
             var vel = (ray.GetPoint(length) - dragObj.position) * speed; 
             // limit max velocity to avoid pass through objects
             if (vel.magnitude > maxSpeed) vel *= maxSpeed / vel.magnitude;
             // set object velocity
             dragObj.rigidbody.velocity = vel;
              dragObj.transform.localScale = Vector3(-1.69,1.19,-0.001);
              dragObj.transform.position.z = -0.6;
              
     }

But when I drag object and in the same time I touch screen my second hand, object is moving to second hand or is moving like crazy. I want to when I click object and drag it with first touch.

example: puzzle game. Child when play and drags object often touch screen second hand . And object go to left hand when he touched the screen by accident