Syncing hinge joint over network

I have a system set up where players can interact with objects. They press the "e" key, and it does a raycast, if there is an object in the "interactable" layer it will attach the object to the player's child, called "hand". The "hand" is an empty game object with a script on it, that looks like this (comments are actual code, but changed to make it easy to read):

 public void Update()
     {
         //set position of hand to be 3 units in front of player camera
         
         if (isInteracting)
         {
                 PickObject();
         } else
         {
             //Destroy the joint, and reset the object's layer/rigidbody
         }
     }
 
     void PickObject()
     {
         RaycastHit hit;
         Ray ray = new Ray(playerCamera.transform.position, playerCamera.transform.forward);
 
         if(Physics.Raycast(ray, out hit, PlayerConstants.interactRange, objectLayerMask)){
             pickedObject = hit.collider.gameObject;
 
             if (GetComponent<HingeJoint>() == null)
             {
                 SetRigidBodies();
 
                 //change layer so other players cant interact with it
             }
         }else
         {
             isInteracting = false;
         }
     }
     
     void SetRigidBodies()
     {
         Rigidbody rb = pickedObject.GetComponent<Rigidbody>();
         joint = gameObject.AddComponent<HingeJoint>();
         joint.autoConfigureConnectedAnchor = false;
         joint.connectedAnchor = Vector3.zero;
         joint.connectedBody = rb;
         joint.connectedBody.useGravity = false;
         rb.constraints = RigidbodyConstraints.FreezeRotation;
         joint.breakForce = rb.mass * 10;
     }

This system works perfect on the host player, but on clients if they pick up and object, it shows on their side for half a second, then the object drops and goes back to where it started (it's not because of the break force, i tested that).

How can I sync this across the system? I tried adding a syncvar to the isInteracting boolean, but that didn't seem to fix anything.