Directional Lights STILL affecting shadowed objects!

It's a very frustrating problem. I have a directional light in my scene which will be rotated during gameplay to simulate day & night cycles.

The directional light (which is set to cast soft shadows) is working perfectly well for exterior of the house. The side of the house that is away from the light is shadowed, and, the correct amount of light pours through the windows onto the modular assets of the interior of the building.

HOWEVER, the modular objects INSIDE the house, which (unless they're infront of a window) should be recieving no light at all from the directional light, still seems to be marginally affected by it when I rotate it.

They aren't lit up like they would be if I set the DirLight to NO Shadows, but they have a glimmer/shimmer to them, which I originally thought was the specular/normals still taking the Dir light into account even when it shouldn't, the the same thing happens if I use a standard Diffuse only shader.

I really don't want to have to set up light layers as I am using DEFERRED rendering and I only have a limited amount of those which need to be used for a far more important feature.

Any else had this problem before?