Click to Move multiplayer

Hello, I'm quite a novice and I'm trying out some things but I've ran in to a problem

So, I'm currently attempting to make a multiplayer click to move style game.

I've made a prefab with it's own network identity and I can join locally and have multiple prefabs running.

But it seems that when I have multiple prefabs with their own Raycast style movement they get confused and try to use all cameras to raycast.

So when I have only one prefab ingame it can move around perfectly fine, but when multiple of them join it starts to cause movement issues.

Each prefab also uses a child camera object that runs its own camera script. But the movement is on the prefab it self.

This is my movement script:

 public class MoveToClickPoint : NetworkBehaviour
 {
     NavMeshAgent agent;
 
     void Start()
     {
             agent = GetComponent<NavMeshAgent>();
     }
 
     void Update()
     {
         if (!isLocalPlayer)
         {
             return;
         }
         if (Input.GetMouseButtonDown(0))
         {
             RaycastHit hit;
 
             if (Physics.Raycast(Camera.main.ScreenPointToRay(Input.mousePosition), out hit, 100))
             {
                 agent.destination = hit.point;
             }
         }
     }
 
 
 
 }