Why does BoxCollider2D distance depend on distance to camera center?

I'm designing a 2D background and foreground style system with one GameObject being the background and lots of other children GameObjects being the foreground. The camera is set to orthogonal and there is a Physics2DRaycaster attached to it.

My problem is that all but the central child objects are not receiving events. When I look at the event system I can see that the distance property is 10.999 for the central object and 11 for my background. When I disable the background BoxCollider2D component all the foreground objects start receiving events and the EventSystem distance property reads progressively larger values based on how far away these objects are from the center of the screen.

This type of behaviour is standard for a 3D raycast but why is it being applied here? How do I work around it? Is my system designed poorly, and if so what would be a better alternative? Also is this a bug or an intended design, because I would think that in an orthogonal projection the distance between the camera and an object should be determined based on their view axis distances rather than a 3D algorithm.