Need help with Quaternions - Making a VR Combination Lock

Alright so this is sort of complicated, Essentially the user is holding a combination lock in one hand (and can only manipulate it in one hand due to hardware constraints). The lock has 2 pieces, the dial, and the outer lock. In this case the lock is facing forward in the X axis, just cause that's how the model came in. The desired experience is if the user rotates the lock along the Z or the Y axis, the whole lock should comply (dial and outer lock.) if they rotate it along the X then the outer lock should stay visually locked in place, and the inner lock should rotate with their hand as if they were turning it. The position of the outer lock and dial should also always be the same as the players hand, so all we really care about here is how they are rotating it.

I have this behavior almost working by tracking the parent's rotation delta in the X and applying the inverse to the outer lock while keeping the dial's rotation the same as the parent object's rotation. BUT if the user rotates the lock more than 90 degrees left or right in the X axis, then the rotation goes bonkers and starts rotating the outer lock in ways that are not anticipated.

Here is my code. It seems like the solution should be dead simple, but I always have trouble getting quaternions/euler angles to cooperate.

 public class CombinationLock: MonoBehaviour
 {
     public Transform outerLock;
 
     private Vector3 lastEuler = Vector3.zero;
 
 
     private void Update()
     {
         Vector3 currentEuler = transform.localRotation.eulerAngles;
 
         float delta = currentEuler.x - lastEuler.x;
         outerLock.localRotation = Quaternion.Euler(outerLock.localRotation.eulerAngles.x - delta, 0f, 0f);
 
         lastEuler = transform.localRotation.eulerAngles;
     }
 }