Single lightmap with day/night cycle

My game requires a day/night cycle. I'm not using any realtime shadows, just a single Beast lightmap on the terrain with ambient occlusion and baked shadows. I want to be able to bake a lightmap onto the terrain with shadows and ambient occlusion, then adjust the directional light color and ambient light color dynamically. This seems to be impossible in Unity. I've tried using Dual Lightmaps (had to make some shader changes to get it to work in Forward rendering), and it worked okay, I just baked the dual lightmaps then set the shadow distance to a really high value so the far lightmap was never seen (acceptable in my case, as I'm not using realtime shadows), but this only gave me ambient occlusion, no baked shadows, and the ambient occlusion was strangely bright and washed out.

I've also tried using a custom terrain shader I found online that allows control over the terrain color, which allowed me to adjust the color in sync with the directional light/ambient light color, but this was clunky and didn't provide very nice results.

And yes, I know that baked shadows + day/night cycle might look strange, as the shadows will never move, but I'm keeping the sun light in a fairly neutral position (casting shadows more or less straight down), and my game is heavily stylized, so it should look fine.

I'm probably overcomplicating this, the answer is probably really simple, but I'm still stuck.

Any ideas?