Quaternion - Euler Angles, weird variable swapping when converting

Hi everyone. I'm making this android game and I want to move my player camera by dragging across the screen. I got a really useful script for it, but I'm having a little problem that bugs me and I can't solve. I've been looking for hours for the solution, but I can't find it. So, here's the deal: I get the difference between my initial mouse position when I click (or tap) the screen and then I get the difference to the new mouse position. After that, I apply the rotation. Everything works fine, BUT, if I click (and hold) on the same spot twice, the camera jumps to a new rotation the first time and backwards the second time. I checked the rotation Quaternion and the X and Y variables swap, making this weird effect. I don't know why it is happening, so If any of you can shed some light on the matter I'd be really thankful.

CODE:

     private bool bDragging;
     private Vector3 oldPos;
     private Vector3 panOrigin;
     public float panSpeed;
     public Vector3 pos;
     public Quaternion auxQuaternion;
 
 
     private float clickTimerDefault = 0.5f;
     private float clickTimer;
 
     private Camera firstPCamera;
 
     new void Start()
     {
         base.Start();
         firstPCamera = GameObject.Find("FirstPersonCamera").GetComponent<Camera>();
     }
     // Update is called once per frame
     void Update () {
 
         if (Input.GetMouseButtonDown(0))
         {
             bDragging = true;
             oldPos = firstPCamera.transform.rotation.eulerAngles;
             panOrigin = firstPCamera.ScreenToViewportPoint(Input.mousePosition);                    //Get the ScreenVector the mouse clicked
             clickTimer = clickTimerDefault;
         }
         if (bDragging && clickTimer > 0)
         {
             clickTimer -= Time.deltaTime;
         }
 
         if (Input.GetMouseButton(0) && clickTimer <= 0)
         {
             pos = firstPCamera.ScreenToViewportPoint(Input.mousePosition) - panOrigin;    //Get the difference between where the mouse clicked and where it moved
             Vector3 aux = oldPos + pos * panSpeed;
             aux = new Vector3(-aux.y, -aux.x, 0); //Thi conversion was needed to adjust the axis
             auxQuaternion = Quaternion.Euler(aux);
             firstPCamera.transform.rotation = auxQuaternion;
         }
 
         if (Input.GetMouseButtonUp(0))
         {
             bDragging = false;
         }
     }

First click

Second click

Please somebody help me :c

Thanks.