Network and NavMeshAgents

Hi, im having a problem synchronizing the navmesh agents over the network.

The problem is, that when the Enemy is starting to chase the player and the Enemys location is being synced over the network, then the Enemy will move kind of flickery or move very slowly, like stays in a one spot for a long time before moving, tries to move but something is like blocking it. When the server is started and there's just one player, the enemy moves fine and smoothly. But when atleast 2 players the Enemy moves slowly and wierd.

Any suggestions how to fix this or how to synchronize nav mesh agents.