Inverse Quaternion data from Kinect

I have a scene set up with a 3D model, I'm receiving XYZW quaternion orientation data from my Kinect sensor and setting it to rotate my skeletons joints.

leftShoulderRig (this is a reference to the joint gameObject)

leftShoulderOrientation (this quaternion object stores the XYZW data)

leftShoulderRigObj.transform.localRotation = Quaternion.Inverse(leftShoulderOrientation);

The problem:

The left shoulder rotates as expected however when I use the same code on the right arm the up an down motion is reversed. I use Quaternion.Inverse to fix the problem of all the axis being in the wrong direction but now I just want to reverse a specific axis.

I dont want to use EulerAngles due to the fact that its not as accurate as Quaternions and Gimbal Locking messing things up.

There HAS to be a way to set the right arm to move down and not up when I move my arm, the same way the left side does.

Thank you for your patience, I did a lot of googling and research before formulating this question.

Please ask if you need further clarification.