Bridge Paradox @ 2D Collider Problem

How do you handle colliders in Unity when you're working in a plane and the problem has a third component?

As you can see in the picture a character would be able to walk up the stiars and over the bridge but there is a problem since the collider is in a plane it also allows a character to walk of the bridge to the sides half way. The same is true for walking into the brigde from the sides. How can I handle this, apart from designing a world where brigde like this don't exist. And I still want character to walk under the brigde aswell as over it.