Scroll view not scrolling when colliders are in the way

Hello. I am having an issue with a basic scroll view setup. It seems that when the content of the scroll view has 2D box colliders (in this case to allow OnMouseOver detection) the mouse cannot "touch" the scroll rect, and therefore it cannot scroll.

Here you can see the layout of "tiles", each having a box collider taking up the full square, and this is where scrolling the mouse button does nothing:

And here, in the blank space with no tiles (and colliders), scrolling works normally:

So, my question is essentially: Is there a way to scroll a scroll view without having to hover directly over the scroll rect without anything in the way? Or if not, is there a way to allow the mouse detection to pass through the colliders on top and interact with the scroll rect below them?