UI button not working (Unity 3D)

I have a weird situation here. I've got UI images with colliders and Rigidbody 2Ds and I've got neat in-UI physics going. I attached a button (as a child) to the image with the physics, but the button doesn't even sense that it's being hovered over let alone clicked. What's going on here?