Problem with scene view auto exposure while in play mode

I recently started encountering this issue while testing some portals between some caves and my terrain where I would enter play mode from each side. Scene view has auto exposure which does help me see a rough estimate of my lighting.

https://imgur.com/ETK67Ho (gif of what's happening)

So everything looks normal while editing, but the moment I enter play mode in my cave area, the scene view tab over-exposes extremely badly, flooding everything in extreme brightness. However, if I enter play mode from my terrain, the scene view tab seems under-exposed, where my cave area is completely shaded over to pitch black. And if I enter play mode between the two, then it's a mix of the terrain being over-exposed and the cave being under-exposed.