Torque rigidbody toward desired rotation on two axes ignoring Y axes?

I am trying to make a Rigidbody disc be spinnable around its Y-axis (upward), while also staying level on the X/Z axis. Currently, the thing works unless I try to remove the Y-axis (by setting it equal to the target Y rotation using eulerAngles.y)

EX:

 // toTrack is a transform & trackerRB is the rigidbody that should copy its rotation
 
 Quaternion rotationToTrack = toTrack.rotation * Quaternion.Euler(rotationOffset);
 Quaternion localRotationToTrack = Quaternion.Inverse(trackerRB.transform.rotation) * rotationToTrack;
 localRotationToTrack = Quaternion.RotateTowards(localRotationToTrack, rotationToTrack, 360);
 Quaternion difference = (Quaternion.Inverse(trackerRB.transform.localRotation) * localRotationToTrack);
 difference.ToAngleAxis(out float angle, out Vector3 axis);
 Vector3 torque = Vector3.Scale(angle * axis, torqueMult);
 trackerRB.AddRelativeTorque(torque, ForceMode.VelocityChange);`

THIS works

BUT if I set the Y rotations equivalent (before finding the difference, as to remove the Y component of the rotation) by using

 Vector3 localEulers = localRotationToTrack.eulerAngles;
 localEulers.y = trackerRB.transform.localRotation.eulerAngles.y;
 localRotationToTrack = Quaternion.Euler(localEulers);

then the entire thing behaves weirdly, trying to flip over, etc., I think it's a Euler angles issue (kinda confirmed because Euler angle (179, 0, 0) prints fine, but the next print out is (0, 180, 180) and so I'm not sure how to remove the y component from the calculation without converting to Euler. I've tried a lot of things but I'm not sure how I can achieve this reliably.

Any help would be hugely appreciated!