Freezing X and Y rotation not working (rigidBody)

Hi,

I'm trying to rotate a gameObject around its Z axis and want to freeze the X and Y axis. I've done this in the rigidBody constraints but after a some time or a few collisions the object starts spazzing out in the frozen axes.

This is the code I'm using for the rotation of the gameObject:

 if (Input.GetMouseButton(1) == true || Input.GetMouseButton(0))
         {
             if (Input.GetMouseButton(1))
             {
                 GetComponent<Rigidbody>().AddTorque(Vector3.back * manoeveringThrust * Time.deltaTime);
             }
 
             if (Input.GetMouseButton(0))
             {
                 GetComponent<Rigidbody>().AddTorque(Vector3.forward * manoeveringThrust * Time.deltaTime);
             }
         }

I'm using AddTorque because this is a space game and I want it to feel floaty and to go with the general theme of the game of using force to simulate gravity and such. The player looks at the game from a 2D perspective so whenever the player starts rotating in the X or Y axes he gets stuck.

Am I understanding/doing something wrong with these rigidBody constraints and is there a way to forcefully freeze those rotations? Ideally I'd want to just force the X and Y rotations to be 0 at all times.