rigidbody affecting other rigidbodies: vibrating objects

Hello unity community,

I'm having an issue with how a non-kinematic rigidbody is affecting other rigidbodies (also non-kinematic). I have a pretty standard scenario: In a first person perspective, I have a model of a desk, with openable drawers, which have items placed inside them. The drawers have a configurable joint (set to mimic a spring joint) to limit their movement to one axis and give them a maximum travel distance, which anchors to the world. Furthermore, when a drawer is clicked, a joint is created between the drawer and an empty gameObject (which functions as a temp placeholder for the hand position).

Everything is fine up until this part, I can pull open the drawers, they stop where they are supposed to, and I can close them smoothly. However, if I place an item with a rigidbody inside the drawer (for example by dropping it in at the start of the scene), as soon as I click and create the joint between "hand" and drawer, the item starts vibrating, to the point that it shoots out of the drawer sometimes. Furthermore, if I open the drawer too quickly, the item disregards the drawer's colliders, and either shoots through the drawer walls, or falls through the bottom. Note that the item inside the drawer is not connnected to anything by joints, and is just resting on the drawer's bottom, by using gravity.

The following code snippet is attached to the drawer gameObject:

 if (Input.GetMouseButtonUp(0) && itemHeld == 1) {
          
          if (this.gameObject.name!="drawer")
         {
         heldObject.GetComponent(Rigidbody).constraints = RigidbodyConstraints.None;
         heldObject.GetComponent(Rigidbody).drag = 1.0;
         heldObject.GetComponent(Rigidbody).mass = 1.0;
         heldObject.transform.parent = null;
         }
         Destroy(hand.GetComponent(ConfigurableJoint));
         
         heldObject = null;
         itemHeld = 0;
         GameObject.FindWithTag("Player").GetComponent(objFocus).handsFull = false;
         activeItem = false;
         }
     
         else if (Input.GetMouseButton(0) && itemHeld == 0) {
             if(plrFoc){
         
                 Debug.Log(this);
                     
                 hand = GameObject.Find("itemHeldPos");
                 var handJoint;
                 if (this.gameObject.name=="drawer")
                 {
                 hand.AddComponent(ConfigurableJoint);
                 handJoint = hand.GetComponent(ConfigurableJoint);
                 handJoint.connectedBody = heldObject.GetComponent(Rigidbody);
                 handJoint.xMotion = ConfigurableJointMotion.Locked;
                 handJoint.yMotion = ConfigurableJointMotion.Locked;
                 handJoint.zMotion = ConfigurableJointMotion.Locked;
                     
                 handJoint.angularXMotion = ConfigurableJointMotion.Free;
                 handJoint.angularYMotion = ConfigurableJointMotion.Free;
                 handJoint.angularZMotion = ConfigurableJointMotion.Free;
                 handJoint.linearLimit.damper = 1;
                 }
         
                 itemHeld = 1;
                 GameObject.FindWithTag("Player").GetComponent(objFocus).handsFull = true;
                 activeItem = true;
             }
             else Debug.Log("too far");
         }

To clarify, the plrFoc is a boolean var in a script attached to the player, to check which object (if any) is in focus and is close enough to be picked up. The configurable joint being created in this script is the one between hand and drawer, and seems to be the one producing the vibrating item behaviour, as when I release the mouseButton (thus releasing the drawer) the item rests again.

Unfortunately, my unity knowledge is not that great, and although I have a feeling that it has something to do with physics and collisions, I have no idea where to look.

Thanks, and sorry for the long post..

PS. I've thought of some workarounds for this, mainly creating loose joints between items and drawer, and breaking them as soon as i touch the object, but I would prefer not to resort to this, as I may have to do this for lots and lots of small objects.