Unity joint physics jitter - Attach a small object (Sword hilt) to another object (FPS camera or player hand)

How would you attach an object to the first-person camera (To the player basically)? To allow the player to carry, move around, and swing an object or a sword.

I've tried fixed joint and configurable joint. I've tried having a 2 joint setup = camera -> hand object (a sphere) -> Object

And having the hand object move with velocity through a script rather than be attached to the camera with a joint, and then attaching to the object with a configurable joint.

The problem is there's a lot of visible jitter in both cases. The sphere moves smoothly with the camera when there's nothing attached. When I attach an object with a joint, it jitters and doesn't move smoothly anymore.

I've tried the common fixes for joint jitter mentioned in the Unity docs.

This also seems to be because of scale - the object, or a sword hilt, would be a thin, small object, not a full-sized 1x1x1 cube. I've tried making the object small through the scale, and also through generating the mesh with smaller custom triangles and quads. Doesn't help.

Any ideas?