Button not clicked when a 2D collider is behind

I have a number of buttons that does not work when a 2D collider is beneath, as shown in the picture below. When i click on the part with white tiles behind nothing happens, but when i click on the lower half the button works as intented.

The button image has raycast target enabled. The canvas containing the button has a canvas group with Blocks Raycast and Interactable enabled. The tiles beneath the button has 2D colliders which is used, however the button should have first priority. The position in the z axis of the two elements does not matter. Have anyone had this problem or have ideas thatat might fix it? I have been looking quite a while for a solution but i cannot find any that works :(