Buttons not always working

I have a math game where a question is at the top of the screen on the camera canvas and 3 answers (1 right, 2 wrong) are on a world canvas in front of a background. The answers have a buttoned child'd to each one (one has it set to the correct answer func, the other two go to a wrong answer func) and sized up properly. During Runtime the buttons randomly decide not to work sometimes. I haven't been able to find any pattern in when they don't work, it just randomly doesn't register me clicking them. I exported to mobile and it has the same problem. Note: This also happens in the editor. Nothing in the scene has any colliders on it.