Adding relative torque on x and y causes z changes

I am just prototyping a quick control system for a space ship and I am trying both relative torque and move rotation however both seem to suffer from the same issue.

So if I were to have something like:

 var deltaRotation = Quaternion.Euler(pitchChange, yawChange, 0);
 _rigidbody.MoveRotation(_rigidbody.rotation * deltaRotation);

I have also used:

 _rigidbody.AddRelativeTorque(pitchChange, yawChange, 0);

I would expect that that the Z axis would not change, but the X (Pitch) and Y (Yaw) would, and it does, but if I were to apply X and Y together it often automatically rolls the ship incrementally. I know I can freeze Z rotation but I would still like Z to have a physical impact from things, so I was wondering is there some way around this problem?