AddRelativeTorque causing wild spining

Hey all, I am trying to make simple flight controls. Using the mouse for roll and pitch and the horizontal axis for yaw.

 void FixedUpdate () {
 
     // Get Input
     float rollInput = Input.GetAxis("Mouse X");
     float pitchInput = Input.GetAxis("Mouse Y");
     float yawInput = Input.GetAxis("Horizontal");

     // Calculate Change
     float targetYawVelocityChange = yawInput - rigidbody.angularVelocity.y;
     float targetRollVelocityChange = rollInput - rigidbody.angularVelocity.z;
     float targetPitchVelocityChange = pitchInput - rigidbody.angularVelocity.x;


     // Apply Torque
     rigidbody.AddRelativeTorque(targetPitchVelocityChange,
                                 targetYawVelocityChange,
                                 targetRollVelocityChange);
 }

I am trying to use a similar idea to the RigidbodyFPSController on the wiki, using the velocity change so that when all input stops the craft will slow to a stop (rotationally)

The problem is this, whenever I turn it starts madly spinning and I cant really see why. Can anyone point out my error?