Unity 5 Audio Source volume falloff for mono sound without panning

I'm looking for a way to fade out Audio Sources depending on their distance to the audio listener, however I want all of my sounds to be mono without any panning. Is this possible with the default Unity components or do I have to write my own script?

It seems very easy to do, because I would think that I could set my volume falloff in the 3D Sound Settings of the Audio Source and just use a mono sound, but it still pans between left and right depending on the position of the source.