collision detection on user-controlled rigidbodies

Suppose I have a reconstructed hand mesh in real time, and this mesh is imported to Unity, and build a Rigidbody and MeshCollider for it in real time as well. So this hand-mesh rigidbody is actually moved by my hand, not the engine, but I need the mesh to interact with other rigidbodies created by Unity, like a sphere. I find that collision detections between the hand mesh and other rigidbodies are always missed. I guess the reason is that the hand mesh is not simulated by physics. Do I need to use a Character Controller for the hand mesh? Thanks.