Box Collider Working Sparadically,Box Collider Sporadically Working

Hey!

I am new to unity.

I am currently using VR Simulator alongside XR Roomscale Rig.

I have created a cube in the middle of a plane which I have set with Rigidbody and a Box Collider (see image one for set up).

I created hands under the XR Rig with a prefab of cubes to represent the hands. The hands work fine with the VR simulator. These hands also have a box collider but not a rigidbody (see the second image). I am finding that when I try to move the square hands to interact with the cube in the middle of the room (by knocking it over), it interacts with the object sporadically. So I am moving my cube hand back and forth, but the hand keeps going through the cube in the middle of the room. Except every now and then the physics kick in and it finally moves. but it goes as quick as it comes.

When I first started this set up it worked where my hands consistently collided with objects but I didn't write down all the steps I took. What am I missing?