Trouble with rotating player when gravity is shifted

Not sure if this is the place to ask because this is a very specific bug, but what the hey.

I'm working on a 3D mechanism in which the player can shoot at a wall to determine it as the new floor, allowing for 6 different "gravities" in the game (the environment is a connected series of boxes). I've gotten it to work perfectly on 4 of the 6 directions, but somehow the front and back walls (the two walls along the global Z axis) give me the strangest trouble.

Below is my cameraController script. The sensitivity\smoothing bit was copy-pasted from a youtube tutorial, but I can't see any fault in it. I've also set constraints on the player's rigidbody, freezing rotation on X and Z axes and leaving only Y free. Removing these constraints doesn't fix anything, it only makes the player sway towards the direction he is rotating and tip over.

     mouseInputX = Input.GetAxisRaw("Mouse X") * sensitivity * smoothing;
     mouseInputY = Input.GetAxisRaw("Mouse Y") * sensitivity * smoothing;
     smoothX = Mathf.Lerp(smoothX, mouseInputX, 1f / smoothing);
     smoothY = Mathf.Lerp(smoothY, mouseInputY, 1f / smoothing);
     mouseLookX += smoothX;
     mouseLookY += smoothY;
     mouseLookY = Mathf.Clamp(mouseLookY, minimumY, maximumY);

     transform.localRotation = Quaternion.AngleAxis(mouseLookY, Vector3.left);
     player.transform.rotation = Quaternion.AngleAxis(mouseLookX, -gameController.gravityVector);

     if (gameController.gravityVector == Vector3.forward) //still buggy!
     {
         player.transform.eulerAngles += new Vector3(-90, 0, 0);
     }
     else if (gameController.gravityVector == Vector3.right)
     {
         player.transform.eulerAngles += new Vector3(0, 0, 90);
     }
     else if (gameController.gravityVector == Vector3.back) //still buggy!
     {
         player.transform.eulerAngles += new Vector3(90, 0, 0);
     }
     else if (gameController.gravityVector == Vector3.left)
     {
         player.transform.eulerAngles += new Vector3(0, 0, -90);
     }
     else if (gameController.gravityVector == Vector3.up)
     {
         player.transform.eulerAngles += new Vector3(0, 0, 180);
     }

The trouble I'm getting is that when, for instance, gravity is set to the front wall (gameController.gravityVector == Vector3.forward). Looking up and down (rotating around Vector3.left which is -1,0,0) works perfectly fine, but looking left and right (rotating around -gameController.gravityVector) rotates the player around his personal Z axis instead of his Y axis. Mind that the problem doesn't happen on the floor, ceiling and side walls.

I've tried conditioning the rotation for each wall separately, and toy with the axis I give the AngleAxis function. This is where I got super stuck, because both Vector3.down and Vector3.back identically rotate my player around his freaking Z axis! (Vector3.up and Vector3.forward are also identical to each other, yielding an inverse rotation around the Z axis).

I am all out of ideas on how to fix this, and am beginning to believe AngleAxis has a bug.

Whoever helps me solve this will earn my undying gratitude, which may very well manifest in the form of a pizza.