VR Physics Hands Rotate In Opposite Direction

I tried to make a working vr physics hand and everything was successful, except for the fact that with how the controllers in unity's xr package is tracked there's a certain point in the rotation where the quaternion will flip from 0.6f to -0.6f causing the physical hand to rotate in the opposite direction to re-align itself. The code is completely fine as you'll see I'm just wondering how to circumvent this issue of the hand rotating in the wrong direction because of the sudden sign change and I've been banging my head at it for about a week or two now. Thanks for reading!

[SerializeField] private Transform controller; [SerializeField] private Rigidbody rigidBody; private void FixedUpdate() { rigidBody.velocity = (controller.position - transform.position) / Time.fixedDeltaTime; Quaternion rotationDifference = controller.rotation * Quaternion.Inverse(transform.rotation); rotationDifference.ToAngleAxis(out float angleInDegree, out Vector3 rotationAxis); Vector3 rotationDifferenceInDegree = angleInDegree * rotationAxis.normalized; rigidBody.angularVelocity = rotationDifferenceInDegree * Mathf.Deg2Rad / Time.fixedDeltaTime; }