My Colliders Sometimes Works and Sometimes Doesn't

I am trying to make a VR game so I tried adding meshes so I can test it out. The colliders for the meshes sometimes work when I put them on Mesh Colliders, the same goes for the box collider. Sometimes my hand would go through it and sometimes it will not, and I can't figure it out.

Thanks in advance!