Why are some of my normal mapped surfaces not being lit?

For example, I have a building that I just modeled and added some nice brick textures (looks really good imo) but when I make the material a Bump Diffuse and add the normal map, the lighting gets really strange. When shining a spotlight at the rear of the building, it doesn't light up pretty much at all. Where as shining the same light at the front of the building looks perfect.

My normal mapped texture is marked as a Normal Mapa and everything, so I'm not sure what to do.