NavMesh giving jerky like motion

I instantiate Gameobjects that have a NavMeshAgent attached to them. Their destination is the player that can move around a map, that has a baked NavMesh on the map.

The problem I'm having is that after a few of these enemy Gameobjects get instantiated, they sometimes stop for a split second, giving a jerky motion, as if they're calculating a path as the player moves round.

Is there a way to combat this? Or would I have to have a more simple NavMesh on the map?

I set the destination of the enemies in Update() and I have also tried in FixedUpdate(), but that makes no difference:

 if (player)
    GetComponent<NavMeshAgent> ().destination = player.transform.position;