Why can't I get pitch black more than just around the camera or myself in the viewport?

Here's the deal:

I want a scene where the ground seem infinite and my character dance in the middle. I've used spots lights to make a threepoint lighting and a directional light to get a shadow on the ground. To obtain the spot light feel with the directional light, I've used a big plane with an hole in the middle. The ambient light is completely black and the background color of my camera is black too. The problem is that the scene look right just at a certain range around the camera. It seems like my plane dont cast shadow everywhere to save ressource or something like that.

Here's some images to illustrate what I'm facing: