Screen to World Point on 2.5D game

Hi guys, I'm having some trouble here with the Screen to World point function. I want in my game to control the position(Only x and y) of a spaceship through the position of the mouse on the screen. So, when I'm clicking, the spaceship would be following the mouse position(obviously using "screen to world point"). It is working really fine but when I stop holding the click button, the spaceship stops immediately and I wanted it to be smooth, like slowing down the speed. The movement is already smooth, cause I'm using Vector3.Lerp, here is the code:

 void FixedUpdate()
     {
             Vector3 desiredPos = LeanTouch.Fingers[0].GetWorldPosition(20f, Camera.main);
             Vector3 smoothedPos = Vector3.Lerp(rb.position, desiredPos, smoothSpeed * Time.deltaTime);
             rb.MovePosition(smoothedPos);
         
     }

So the LeanTouch.Fingers.GetWorldPosition is basically ScreenToWorld, the same thing, I tested both. I guess using AddForce instead of MovePosition would fix that but when using it then my object gets crazy and goes to a random direction. I tried a few solutions on the internet but they didn't worked, I saw one about making a raycast but I didn't get it quite right. Does anyone know a good solution?