Manipulate the Global Ambient light in a shader?

I want to write my own shader that doesn't use the GI solutions in Unity (not baking lightmaps, nor the Pre-computed Real-time GI), I only want to use the Gradient or Ambient Color in the Environment Lighting.

What I want to do is to multiply the Ambient color with the vertex color of the mesh (the vertex color is the baked ambient occlusion from the 3D software I make the mesh in), before Unity adds the lighting together.

I have tried this method, using the unity_AmbientSky and multiply it with the vertex color from the Input struct and adding it to the Albedo.

 void surf (Input IN, inout SurfaceOutputStandard o) {
     fixed4 c = _Color + unity_AmbientSky * IN.vertexColor.r;
     o.Albedo = c.rgb;
 }

The problem with this method is that I don't really modify the lighting, but rather I manipulate the Albedo color, and if so I might as well just paint the lighting and AO into the albedo texture from the start. Or do I missunderstand how the void surf() function actually works? I mean where in the shader does unity do the lighting stuff? A very basic Surface Shader does automatically handle light, shadows and GI/ambient without I having to add or multiply the Albedo with the lighting, so where does a Surface Shader do the "lighting stuff" and how do I interfere with it? Because I don't want to take the Albedo color, add the Ambient color multiplied with the vertex color and then Unity adds the Ambient color again as part of it's built in lighting pass. But I don't know how the shader pipeline works here. But my point is: I don't want to change the albedo color of the material, I want to manipulate the global Ambient color before it is added to the final screen pixel color. Check out this blog post about how to multiply the AO pass in post-production when rendering with offline renderers (like mental ray) The Joy of a little "Ambience"... it might explain this a bit clearer, with image examples.

I want Unity to take care of the direct lighting, and using Surface Shader (rather then Lambert, or Unlit) for taking care of shadows, reflection, specular, emission, and so on.

I'm mostly want to use this for mobile and VR, so I suppose the Forward Rendering path is the one I'm going for, or the new Scriptable Render Pipeline in Unity 2018.