Agent rotates when destination is set, causing it to get stuck and not move.

Unity's NavMesh has been giving me so many issues that I'm about to just write my own solution, but I'd still rather avoid that if possible. Right now, I'm having an issue where agents will rotate before moving, but only under specific circumstances (when standing on an unreachable part of the navmesh, or after warping the agent). Please help, I'm completely stuck at this moment and members of my team are waiting on me. Demonstration As you can see in this video, the cube (being an agent) is fine following the player normally, but as soon as I jump up onto the other platform, it starts looking back every time I set it's destination to the player.

 using System.Collections;
 using System.Collections.Generic;
 using UnityEngine;
 using UnityEngine.AI;
 
 public class Test : MonoBehaviour
 {
     NavMeshAgent agent;
 
     void Start () 
     {
         agent = GetComponent<NavMeshAgent>();
     }
 
     void Update()
     {
         if(Input.GetKey("e") && !agent.pathPending)
         {
             agent.destination = new Vector3(Player.localPlayer.transform.position.x, transform.position.y, Player.localPlayer.transform.position.z);
         }
     }
 }

I made a script as simple as I possibly could to make sure it wasn't my enemy component's fault and it still happens.

This is what the NavMesh looks like.

Thank you! :)