Server controlled Joint in UNET

Hi, I have this little multiplayer test game where players can shoot off each others hat (s. picture). The Hat has a Rigidbody Component and is attached to the Capsule through a Hinge Joint. My goal is to fully let the server decide (it's a game with a host and no dedicated server) about when the joint is breaking and the hat falls off. Therefore I've written the following script:

 public class NetworkPlayer : NetworkBehaviour {
 
     HingeJoint thisJoint;
 
     public override void OnStartLocalPlayer()
     {
         base.OnStartLocalPlayer();
         gameObject.FindComponentInChildWithTag<MeshRenderer>("Body").material.color = Color.blue;
         
     }
     public override void OnStartClient()
     {
         base.OnStartClient();
         if (!isServer)
         {
             thisJoint = GetComponentInChildren<HingeJoint>();
             thisJoint.breakForce = Mathf.Infinity;
             gameObject.FindComponentInChildWithTag<Rigidbody>("Hat").isKinematic = true;//Extension method which works 100%
         }
     }
 
     public override void OnStartServer()
     {
         base.OnStartServer();
         thisJoint=GetComponentInChildren<HingeJoint>();
         thisJoint.breakForce =10;
     }
 
     
     pulic void JointBreak() //gets called from the OnJointBreak(float breakforce) method from the Hat Children object
     {
         if(isServer)
             RpcDestroyJoint(gameObject);
     }
 
     [ClientRpc]
     void RpcDestroyJoint((GameObject NetworkGO)
     {
         Destroy(NetworkGO.GetComponentInChildren<HingeJoint>());
     }
 
     //....more shooting methods which are not relevant to the problem
 }

Only on the server the joints are breakable. If the joint breaks I intend to let the server send a ClientRPC to all Clients to Destroy the Joint. All Players have NetworkTransform as well as NetworkTransformChild for the Hat (s. picture). If I shoot the server hats it is syncronized on both game instances. The other way around, if I shoot the clients hat only on the server game instance you can see the hat fall of (for some reason the HingeJoint didn't get destroyed on the client game instance). The Players were autospawned by the NetworkManager.