AddTorque/AddRelativeTorque inaccurate

I am trying to work on controls to rotate a spaceship. The idea is to use the input for determining the "desired angular velocity" and then the system applies the necessary torque to reach that velocity. Eventually a PID(for this probably just P)-controller is supposed to take over this task.

My problem is now that AddTorque and AddRelativeTorque seem to work very very inaccurately. Applying a torque around the forward axis with AddRelativeTorque will not result in a clean rotation around that axis but eventually turns into a very unclean rotation that has some yaw and pitch movement in it as well. I get those yaw and pitch velocities from rigidbody.angularvelocity and I can also see it when I ask transform for the forward vector... it changes... so no wonder the rotation gets screwed up. Why does a clean torque around the forward axis end up changing that specific axis?

This becomes incredibly hard to control, not even mentioning the fact that making a PD-controller for that becomes next to impossible. The feedback that I get from transform and rigid body seems to be not accurate enough and the physics system doesn't apply forces accurately enough.

Am I doing something wrong or can't this be done at all? This would be very unfortunate because if I can't move and rotate objects precisely by applying forces to them, then I can pretty much forget about the entire physics system for my game.

Any input on this?