The directional lights color visible in shadows

I've tried finding the answer to this with no success. The thing is that my directional lights color pops up in corners and on edges where there are shadows and where it's not supposed to be. Here's a screenshot:

Anyone know how to fix this?