Help with using joints to recreate a mechanical arcade machine (Zeke's Peak)

I'm trying to recreate something similar to Zeke's Peak (demo on youtube) in Unity, as a multiplayer game.

To summarise, on the left and right edge of the play area, there is a pole with a device that can move up and down the pole at the players control. Each side controls independently. Then there is a beam that runs across the gap connecting the two devices. As the two sides move, that beam naturally rotates to form the line between the two.

My initial thought was to have 2 game objects kinematic rigid bodies for the sides, and then a non-kinematic rigid body with 2 hinge joints for the beam, with each hinge connected to one of the two rigid bodies on the sides.

This worked ok in singleplayer, but the moment I try to make it work over a network (using Mirror), the beam just lags out of sync constantly, ruining the effect.

Any thoughts on a more robust/reliable way to implement this effect?