ShadowCaster2D is not compatible with Perspective Camera

I am making a 2D/2.5D game using Unity's dynamic lighting and I want to add dynamic shadows (like in Hollow Knight), however, the shadows only work when using an orthographic camera. But when using a perspective camera it does generate shadows, except they are not to scale with the screen.

What it should look like: What it looks like:

You can see that the shadow seems to be rendered correctly, just scaled down. Experimenting with the Camera's Z position changed the scale of the shadow.