Weird navigation issue

I am trying to get my agent to move around the NavMesh by clicking on a desired position. It works fine for the most of the mesh, but at a certain threshold the agent starts stuttering and bumps into an invisible wall of sorts, refusing to go past it.

This is how the navmesh looks in the editor:

The walls and the furniture are assigned as non-walkable Navigation Statics, the floor pieces of different rooms are separate planes.

This is what happens in the play mode:

At the point where the character is now he hits the "invisible wall" and starts sliding down towards the lower wall while still oriented to the position I selected (the room to the left). Here is the code I use to tell the agent to move (pretty much copied from the manual's article on my desired form of navigation):

 public class PC : MonoBehaviour {
     public GameObject selection;
     bool selected;
     UnityEngine.AI.NavMeshAgent agent;
     // Use this for initialization
     void Start () {
         selected = false;
         agent = GetComponent<UnityEngine.AI.NavMeshAgent>();
     }
     
     // Update is called once per frame
     void Update () {
         //movement
         
             if (Input.GetMouseButtonDown(1)) {
                 RaycastHit hit;
 
                 if (Physics.Raycast(Camera.main.ScreenPointToRay(Input.mousePosition), out hit, 100)) {
                     agent.destination = hit.point;
 
                 }
             }
         }

I tried changing the camera mode to perspective to see if my orthographic one's depth may cause the issue and removed the UI from the screen in case it may have been blocking the raycast (even though it shouldn't have been since it's screen overlay), nothing changed. Rearranging the scene didn't fix the problem either.