Vector3.MoveTowards and NavAgent weirdness

Hi, this is a repost from this forum thread. Just thinking this is a more appropriate place to post it. I'm having trouble with weird motions coming out of Vector3.MoveTowards and the NavMeshAgent system.

Here's the setup: I have a player gameObject with a NavMeshAgent component; you can click on the terrain to set the agent's destination. And then I have a child gameObject the position of which is some distance between the player's current position and the agent's destination. The idea is that this object will be a focal point for the camera, and it leads ahead the player in the direction it's travelling. (Cinemachine can only do this if it's allowed to rotate, which I don't want). I want this focal object to gradually move around, instead of just snapping around as new destinations are set, so I have it moving with Vector3.MoveTowards. (I have also tried Vector3.Lerp and Vector3.Slerp and they all do the same thing).

I've attached a screen cap which I slowed down to 5fp for clarity. The green line is the player's forward direction, the solid red disc is the NavMeshAgent's destination, and the small blue circle is the focus point I am concerned with.

(The gif is too big to attach here, but you can see it on this forum post).

You can see that it starts out okay, but when the player turns around it goes bananas and gets behind the player, and then it straightens out again. But I don't reference the player's direction anywhere - all this object knows is the player's position and the destination position, so it shouldn't be affected by the player's facing direction at all. So I guess my question is: why is this game object being influenced by the player's transform rotation, if it only knows the transform position of the player and the agent destination?

Here is the code I am using to set the position of the focus point. I call this in Update. (I am clamping the desired point so it doesn't get too far away, because the camera will follow it and the player will then be offscreen).

 private void SetPosition()
 {
     destinationDirection = Vector3.Normalize(agent.destination - player.transform.position);
     destinationDistance = Vector3.Magnitude(agent.destination - player.transform.position);
     desiredFocusPoint = player.transform.position + Vector3.ClampMagnitude(destinationDirection * destinationDistance, clampMagnitude);
     transform.position = Vector3.MoveTowards(transform.position, desiredFocusPoint, moveTowardSpeed * Time.deltaTime);
 }

Thanks for your help!