UI Buttons not always working on Android?

This issue is really bothering me. My buttons work perfectly on pc, but not always on Android.

I realized they are not responding when you tap them and swipe. Even if it's a slight swipe that you make unconsciously.

I couldn't find any similar threads so I wonder if this is just happening to me. But I can't imagine what could I possibly do wrong, since the button system is really simple...