NavMesh: How is the "Not Walkable" layer affecting Walkable terrain

So we have our game terrain.

It's a unity terrain with changing height and a certain path we allow our NavAgents to traverse.

On the terrain there are rocks, which are not painted by the terrain detail mesh feature, and are marked as "Not Walkable" layer.

On the terrain we also have a big mesh renderer/collider, marked as "Not Walkable" (used to prevent avatars from leaving the path as long as we don't use pathfinding. This mesh, is much higher the the terrain.

Once we bake the navigation mesh, while the rocks produce wholes in the terrain nav mesh, the big mesh only cuts through the nav mesh, leaving a lot of redundant nav mesh data intact (unreachable nav mesh of the terrain).

I am not sure what are the performance impacts of this unreachable areas, and I believe it takes unused memory.

Why is this happening? Why our we left with these unreachable areas, and what can we do to avoid that (we are using unitu 3.5.7, and don't have Nav Obstacles present in Unity 4, so that is not an option).

Thanks.