Toggling Ambient Light intensity

I have a weird issue with the ambient light.

In my scene player is able to turn off lights, for this I switch the skybox material to null and the intensity multiplier to 0 so it's completely dark. When I turn the lights back on I put the default-skybox material back on and the intensity multiplier back to 1 but the scene looks like it's still 0. When I select an object in the scene view while in play mode / hover with the mouse above the inspector window/tab then suddenly the ambient turns back on to look like 1 and the scene looks like before turning the lights off, like it should.

Making a build doesn't solve this issue.

Any help for fixing this problem?