Rotate camera around pivot not working on certain angles

Hello, this's my first post here; I hope to ask my question properly.

In a scene I'm creating I want the user to be able to rotate and tilt his camera (main one) around the point where he has made a left click while he's pressing the left click button. So I get the point where the user has started to press with this function:

 void GetPivot()
     {
         Plane plane = new Plane(Vector3.up, Vector3.zero);
         Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
         float rayDistance;
         if (plane.Raycast(ray, out rayDistance))
         {
             pivotPoint = ray.GetPoint(rayDistance); // This's a vector3
         }
     }

It works fine. The pivot point's set just once (everytime the user press the left button) thanks to this piece of code that I wrote inside the Update method:

         if (Input.GetMouseButton(0))
         {
             if (!helding)
             {
                 GetPivot();
                 helding = true;
             }
             RotateAroundPivot();
         }
         else
         {
             Cursor.visible = true;
             helding = false;
         }

After I've the pivot point, Update calls the RotateAroundPivot method that looks like this:

 private void RotateAroundPivot()
 {
     float pivotSpeed = 200;
 
     float horizontalMovement = Input.GetAxis("Mouse X") * pivotSpeed;
     transform.RotateAround(pivotPoint, Vector3.up, -horizontalMovement * Time.deltaTime);
 
     float verticalMovement = Input.GetAxis("Mouse Y") * (pivotSpeed / 2);
     transform.RotateAround(pivotPoint, Vector3.right, verticalMovement * Time.deltaTime);
  }
 

And then the problems arises. While the rotateAround with Vector3.up works like a charm:

*The red cube location equals to the pivot location; it's just for clarity.

... rotateAround with Vector3.right as pivot only works on certain positions:

Basically it only only works fine in the initial camera position that, by the way, it's:

 public Vector3 defaultCameraPosition = new Vector3(0, 11, -24);
 public Vector3 defaultCameraRotation = new Vector3(25, 0, 0);

It also works fine in the opposite side.

I guess this's quite of a noob question, related to the angles control that I should be doing. I noticed that the solution might be in controlling the y camera rotation angle. It seems to be working in the [315º-360º] and [0º-45º]... but I don't fully onderstand why. I really hope someone overhere can give me an explanation and luckily an answer.

Thank you very much.