AddTorque vs Colliders

         float h = Input.GetAxis("Mouse X");
         float v = Input.GetAxis("Mouse Y");
         
         float a = movement.turnSpeed;
         
         float vForce = v * -a;
         vForce = Mathf.Clamp(vForce, -a, a);
         
         float hForce = h * a;
         hForce = Mathf.Clamp(hForce, -a, a);
         
         rigidbody.AddRelativeTorque(vForce, hForce, 0);

I use this to rotate the player ship, but the moment I add a collider the whole thing gets screwed up with uneven pitch/turn ratios.

Is there a way to make the AddTorque ignore the collider or make the force applied scale with the collider dimensions/axis?