Terrain Baked and Realtime lighting

Hello all,

As of now, I have a terrain with some buildings and enemies on it. I would like to bake the building shadows onto the terrain while having the enemies and player be real-time lighting.

I can bake the shadows just fine, along with get real-time shadows, but the lights bake to the terrain and are real-time to allow for shadows to cast onto them , resulting in my terrain to be lit up super bright. How can I fix this?

Quicker the better, its for my finals!!