Navmesh agent got fused in very crowded situation when colliding.

I'm trying to make a crowded crossroad simulation where two pedestrian groups crosses the road by following one guide for each group. First what I have done is creating a simple scene using capsule object to as navmeshagent as you can see from my video: [link text][1]

The problem is, if you see at 0:46 the agent fuses and pops out repeatedly on the track. The destination of the agents is their respective leader while the leader itself have a determined destination. The code for the pedestrian can be seen below:

     using UnityEngine;
     using System.Collections;
      
     public class FollowerAgentMovement : MonoBehaviour {
     [HideInInspector]public Transform NPCLeader;
     NavMeshAgent Agent;
     void Start () {
     Agent = GetComponent<NavMeshAgent> ();
     NPCLeader = GameObject.FindWithTag ("LeaderAgent");// each group have different leader, in the implementation there are 2 tag LeaderA and LeaderB
     }
     void Update () {
     Agent.SetDestination (NPCLeader.position);
     }
     }

And this is the code for the guide(leader)

 using UnityEngine;
 using System.Collections;
  
 public class AgentMovement : MonoBehaviour {
     public Vector3 Destination;
     NavMeshAgent Agent;
  
     // Use this for initialization
     void Start () {
         Agent = GetComponent<NavMeshAgent> ();
     }
    
     // Update is called once per frame
     void Update () {
         Agent.SetDestination (Destination);
     }
 }

I'm also pretty aware of the collider and rigidbody effect as stated in http://forum.unity3d.com/threads/need-advice-or-recommendations-with-navmesh-and-agents.130012/

Since I need to detect collision between agent and based on http://docs.unity3d.com/Manual/CollidersOverview.html the collision action matrix of Kinematic Rigidbody Collider cannot trigger the OnCollisionEnter function so I keep using non-kinematic collider as child of the capsule with rotation and position constrain for the rigidbody to detect the collision and the force didn't affect the navmeshagent component. I also tried removing the rigidbody so it's only the navmesh component that affecting the agent but the fuse problem between agent still exist.

Please someone help me, why this phenomenon happen and how to prevent it? Note: sorry for my bad English. VIDEO LINK [1]: https://youtu.be/G7NVwlpZ544