How to stop pushing a NavMeshAgent?

I have a couple of NavMeshAgents that are roaming around the scene and I have a CharacterController. The agents do their best to avoid the player, but there are times where I catch up and I'm able to push the NavMeshAgents as if they were balloons. How do I stop this? I don't want to be able to push agents around.