Game Object seems to move on it's own

I have a GameObject that I am moving along a baked terrain. I do this by clicking, setting a'destination' object where I click, and then telling the player object to move to the destination. I am now trying to get the object to "warp" from one location on the terrain to another, but when I do this it briefly appears in the desired location and then warps to a random location. I have posted a video here to show the behavior: https://youtu.be/WtmzBLMi9FE

To "warp" the object, I am modifying the transform.position by changing the X and Z coordinates to the desired location and then using a Physics.Raycast for the Y coordinate:

 mainPlayerObject.transform.position = GetRandomStartingLocation(StartLocation.transform.position, -10f, 10f);    
     Vector3 GetRandomStartingLocation(Vector3 pos, float min, float max)
     {
         RaycastHit hit;
         int attemptCount = 0;
 
         while (attemptCount <= 10)
         {
             float x = UnityEngine.Random.Range(min, max) + pos.x;
             float z = UnityEngine.Random.Range(min, max) + pos.z;
 
             Vector3 position = new Vector3(x, 0, z);
             //Do a raycast along Vector3.down -> if you hit something the result will be given to you in the "hit" variable
             //This raycast will only find results between 'min' and 'max' units of your original "position"
             if (Physics.Raycast(new Vector3(0, max, 0) + position, Vector3.down, out hit, max - min))
             {
                 return hit.point;
             }
             else
             {
                 Debug.Log("there seems to be no ground at this position");
             }
 
             attemptCount++;
         }
 
         return pos;
     }

As I mentioned, this code moves the object to where I would like it for a moment and then it moves to a random location. I originally put breakpoints all throughout my code to see why this was happening, but I haven't been able to discover the culprit. I eventually bought this debug tool from the Asset Store. It is a tool that keeps track of where transforms are being manipulated. I ran a test where I was continually sending my object to the desired location and it showed that the position was being changed continually by an unknown script (possibly Unity Rigid Body or Physics code). I have attached an image with the results I don't know if I'm doing something wrong or if I have a misunderstanding of Rigid Bodies, Baked Terrains, or something else, but something really wants my object to be in a position other than where I'm telling it to be. If you have any ideas, please let me know. Thanks!