Top-down click to move with Rigidbody

I've been working at this for a few days now but just can't find a solution. I'm trying to create a top-down click to move Rigidbody controller that will work properly with the character moving along a 3d world (up or down slopes, steps, and falling off objects).

If the player is holding left mouse button the controller should move the entire time the mouse button is held down. If the player simply clicks once, store the location in a Vector3 and move the player to that location.

I'm able to get this to work for the most part using a Raycast to detect where the player clicked, then applying force in that direction (Player's position Vector3 subtracted from Target position Vector3), and everything seems to work properly up until I let go of the mouse button. I can't seem to find a good way to determine when to stop applying force as soon as the player has reached his target.

Right now I'm working with something like this:

 if(IsMoving() && grounded)
 {
     rigidbody.AddForce(velocityChange, ForceMode.VelocityChange);
 }
 
 private bool IsMoving() {
     Vector3 a = new Vector3(moveToHit.point.x, 0, moveToHit.point.z);
     Vector3 b = new Vector3(rigidbody.position.x, 0, rigidbody.position.z);
 
     float curDistance = Vector3.Distance(a, b);
 
     if(curDistance >= 0.2)
     {
         return true;
     }
     else
     {
         return false;
     }
 }

But this doesn't seem to work very well. It' unlikely I'll ever catch curDistance at exactly 0.0 in a single frame, and when the player is falling down or going up ramps things seems to bug out.

Any support on this is greatly appreciated. I'm also open for going at this in a completely different direction, such as with a CharacterController (which I tried before this Rigidbody, though had issues getting my gravity to work out correctly)