IsTrigger Active and Environmental Walls

I'm relatively new to the whole game design and development thing so I'm not sure how to solve most of these problems without immense Googling. I'm making a hide and seek multiplayer based game and I'm building the prototype. In the scene, I have the Hider and the Seeker. My code is pretty basic in that each GameObject can be controlled with keys and move around the space I've built. I have an OnTriggerEnter method active to destroy the Hider if the Seeker touches them. Currently, the Hider's IsTrigger is set to true to check for this collision. My issue is that my test environment has some basic walls around it to prevent the players from the leaving the area. Because IsTrigger is set to true on my Hider, it can easily phase through my environment, which all have colliders as well. The Seeker is fine but I need to figure out how to fix this. I can't disable IsTrigger on my Hider as that would nullify my code and not achieve what I need. Sorry for the essay but figured I'd give as much detail as possible. With me being new, there's prob something simple I am missing but any help will be appreciated. Thanks