Moving Directional Light Shadow Flickering

I have spent half a day trying to figure this out and just can't seem to figure out what is happening.

After about 30 seconds of clicking 'Play' the shadows begin to do this.

Shadow 1: https://i.gyazo.com/603a32c34c12f4d52058d6d19431640d.gif

After 45 seconds, the shadows get worse and do this.

Shadow 2: https://i.gyazo.com/b12426ded48d831c8a0a7844eec9d28f.gif

[preview of external content removed for GDPR compliance as it was including 3rd party cookies]

What have I tried?

• Changing around the 'Shadow Distance'

• Changing the 'Shadow Cascades' to none, two and four and playing around with 'Cascade splits'.

• Changing camera clipping from 0.1 to 0.4, 0.5, 1.0, 2.0 and max clipping to 10,000, 1000, 500.

• Changing camera rendering mode to 'Deferred'.

• Changing Directional Light shadows from soft to hard and the bias settings.

• Changed the lighting to 'Linear' lighting in the 'Player Settings'.

What's in the scene?

• Town is made of mesh colliders and all scene's models are a scale of 1.0.

• EnviroSky system has a directional light moving up and downwards based on time.

Either way, I believe it's nothing to do with the EnviroSky system, but more based on the directional light of the system of how it's moving around. I really don't know what is going on anymore, is this a bug I am dealing with?

What Unity version am I on?

I am currently on Unity 5.6.0f3.