Kinect character throwing ball using Physics

I have a character that is controlled by a Kinect V2 and I want to make that character throw a ball, where the ball reacts to physics. I have seen many questions that touch on some of this, but no complete answer (and searching on Character, Physics, throw, ball, kinect gave 44000+ results).

What I have now is one RigidBody on the characterContainer that is holding my character (its bones and its meshes together).This RB has a mass of 80. I have the ball as a child of the character's right hand, with a RB with mass 1 and a fixed joint to the characterContainer's RB. Both RBs have useGravity set to true, both have isKinematic set to false (obviously, otherwise it will not respond to physics).

Now when I play the scene the ball gets detached from the character's hand. The character moves it's root position relative to the location of the Kinect, but also if I disable updating the character's root position the ball gets detached from the hand. And eventually after a minute or so the character falls through the floor (which it doesn't do in the same scene without the ball).

I also tried to give the right hand a RB and then set the ball's FixedJoint to that RB, but then the ball and hand fall to the floor stretching the character's arm. And I have an experience from an earlier proect that if I add RBs to all the character's bones the character starts behaving erratic because the RBs start colliding with eachother.

I hope someone can point me in the right direction.

The project is quite big (I exported the scene to a UnityPackage and it is 42 MB), so if you need any specific information to help troubleshooting, please let me know.

Thanks!