Performance with high number of background objects: CollectShadows expensive when no shadows are cast

In my current game, I have around 20.000 instances of an object at the background, always visible, with the same material. These objects, which are simple static quads, are all generated during runtime and their texture is simple and low resolution (shader used is Unlit/Transparent due to some corners needing transparency). The original object and its instances are all set to not cast nor receive shadows (they all do not use light probes and and have "reflection probes" off).

However, one of the greatest bottlenecks in my performance right now is in the following Drawing phase of my GPU according to the profiler: Render.OpaqueGeometry -> RenderForwardOpaque.Render -> RenderForwardOpaque.CollectShadows. Testing further, I noticed that in fact, when I either disable all these background objects, or the lights in the scene, such shadow collecting drops from around 8ms to almost zero.

My guess is that what is going on is that the engine is doing some shadow calculation regarding shadows in relation to all those thousands of simple quads in the background. What I do not understand is why that is the case if they are all set to not cast and not receive shadows. How could I proceed to make the CollectShadows skip these thousands of background objects altogether, once and for all?

PS: I tried other shaders, like the standard with no texture. The problem stays exactly the same