Make EventSystem ignore a collider

I have a scene where there are two objects: One invisible object with a box collider, and one cube with a box collider, that has a component that implements IDragHandler, that is placed inside the first collider.

When the cube is placed outside the first collider, dragging works, but when the cube is placed inside the first collider, the first collider eats up the event from EventSystem. Is there a way to ignore the first collider and/or forward the drag event to the object inside the first collider?

Placing the first object in the IgnoreRaycasts layer had no effect.