RenderTexture -> Texture2D using RenderTextureFormat.Depth

There's plenty of information to go from rendertexture to texture2D

 camera.Render();
 RenderTexture.active = camera.targetTexture;
 Texture2D cache = new Texture2D(camera.targetTexture.width, camera.targetTexture.height, TextureFormat.ARGB32, false, true);
 cache.ReadPixels(new Rect(0,0,targetTexture.width,targetTexture.height),0,0), false); 
 cache.Apply();
 RenderTexture.active = null;

Now I made cache a public variable so I can preview the texture in the inspector (so I didn't use Texture2D cache but just the member variable cache in my actual code)

and it totally works, until I turn my RenderTextureFormat into Depth, which is what I want because I only need the depth buffer. Preferably an accurate one, using the depth format. I use this render texture in a shader as well and it contains the information I need, the GPU can read data from it just fine, but this conversion just breaks as soon as a render texture is not a standard render texture.

Alternatively I guess it would be possible to use a normal render texture but then I need to know how to render the Z buffer manually into the RGBA channels with a replacement shader.

Any thoughts on this will be greatly appreciated!