Spatial audio in isometric game

I have to introduce ambient audio to our isometric game, whose orthographic camera moves at X and Y with fixed Z. (Camera's orthographic size is used for zoom in/out.) Various terrain elements like buildings are placed in different Z in the world to handle Z order/overlap scenarios. Now the problem is, since we can have a single audio listener per scene, suppose it is added as a camera's child, it will have a fixed z value and the audio sources attached to the terrain objects will have varying z values. So even if the listener is directly aligned over the terrain object in X and Y, it cannot listen to the source since they are too far in Z axis. Bringing the listener closer to object A in Z axis will cause the same problem for another different Z-Valued object B.

Example of what I want: There is a Workshop playing looped construction sounds. As I pan the camera to bring the workshop in view, the sounds should fade in, sound should be max at workshop zoomed in and centered, and fade out as the workshop is panned out of view.

This is simple to achieve in a full 3d environment where the player object has the listener. But how can one go about this problem in an isometric scenario?

Checked this thread but no solution here either https://forum.unity.com/threads/3d-sound-with-orthographic-camera.389646/