EventSystem sending event to further object

I am using 2D sprites for a game in Unity 5.1, and using the event system for clicking using the touch input module and a Physics 2D raycaster.

I have a background that implements IPointerDownHandler and objects in the foreground that implement IPointerClickHandler and IPointerDownHandler. When I click one of the foreground objects, it sends the message to the background. When I select the event system to see the debug output in the inspector, it keeps the background as the current raycast object when I hover over the foreground objects. It does change the index from 0 to 1, so it is detecting that there is another closer object there, but it never sends the message to that closer object. If I check the index provided on the OnPointerDown function, I can easily ignore background clicks properly when the mouse is over a foreground object. Is there away to force reset the raycast so it uses the foreground objects as the select objects when the mouse goes over them? Alternatively, Is there a way to access the array of objects the returned index is for, so I can manual send the event on to the correct object?

If I disable the background box collider, clicking on the foreground objects works just fine. The background is a good 5 units behind the foreground objects, so when you switch the scene to a 3D view, there is a ton of space between the background and foreground objects and their BoxCollider2Ds.