NavMesh Collision Detection?

I admit the title may be a bit misleading, but what I'm looking for is a feature similar to that of a rigidbody.

I have certain objects in my scene that are NOT (!) supposed to be NavMesh obstacles. They are however supposed to be solid walls. That may sound silly at first but it has to be that way, because NavMesh Agents are supposed to try to walk through that wall but detect it (using a raycast) and then stop in front of it and try to break it down.

That script works pretty fine as long as there is only one NavMeshAgent, but as soon as another one comes by, it pushes the current one through the wall. I want to avoid that. I tried applying a rigidbody, but that only caused my agents to go nuts (they were going all over the place).

Any tips?