Why doesn't my AI collide with walls?

I made an AI using NavMesh and NavMeshAgent. The AI works okay, however even though all my walls and such have colliders, as does the enemy, the AI seems to make him ignore his collider and just walk through walls. I have tried toggling "Height Mesh" in the nav's bake menu, that didn't make a difference. I've spent two days trying to figure this out. I've even removed and re-added colliders and redone the bakes a few times. The AI colliders with me properly, but for some reason it totally ignores the wall, while I collide with it fine.