Smaller object jumps around bigger when dragged by mouse cursor

Hello dear all users,

I have a problem with moving object with mouse cursor.

 private Ray eyeRay;
 private RayCastHit hit;
 
 public void Update(){
             if(allowMouseMoveUnit){
                 eyeRay = Camera.main.ScreenPointToRay(new Vector3( Input.mousePosition.x, Input.mousePosition.y,0) );//;)
                 if(Physics.Raycast(eyeRay, out hit, 1<<8)){
                     transform.position=new Vector3(hit.point.x,1.8f,hit.point.z);
                 }
             }

When scene is empty capsule is moving under mouse, same when capsule is bigger than other objects on the scene. While there are bigger objects on the scene than capsule, and mouse is over them (bigger ones), capsule is jumping around or on them.

Usually all objects have colliders and some have Rigidbody etc.

How to fix this? I would like capsule to move into the other objects when they are bigger, like a "ghost object".

Thank you for time.