How to make camera rotate spherically around an object

I'm trying to create a third-person camera controller that rotates the camera around the player at a fixed distance - think of a camera system like The Witcher 3's, or MGS V's for basic camera movement.

I have rotation around the Y axis working just fine, but rotation around the X axis is more of a problem. The camera doesn't rotate around the player at the widest circumference point, as it would if it move directly upwards when rotating, rather it loops in a smaller circle, depending upon it current Y axis rotation. It's hard to describe without a visual, but try out my code to see for yourself. I think it might be because I don't want to just rotate around the X axis, but rather the X and Z axis', depending upon my Y rotation. I haven't solved it yet though.

This is my camera code:

 public class CameraController1 : MonoBehaviour {
 
     public GameObject pointOfFocus;
 
     public float camRange;
 
     public float rotationSpeed;
 
     private float xRotationOffset, zRotationOffset;
 
     public float minRSSensitivity;
     
     void Awake()
     {
         transform.position = pointOfFocus.transform.position + new Vector3(0f, 0f, -camRange);
 
         //When the pointOfFocus inputs are correct to move the camera
         InputManager.cameraMove += MoveCamera;
     }
 
     void OnDestroy()
     {
         InputManager.cameraMove -= MoveCamera;
     }
 
     //Only gets called when pointOfFocus messes with left stick
     private void MoveCamera()
     {
         //check that the pointOfFocus input is strong enough to warrant cam movement on xRotationOffset
         if(!InputManager.rsXAxisInput.IsBetween(-minRSSensitivity, minRSSensitivity))
         {
             xRotationOffset = InputManager.rsXAxisInput * rotationSpeed;
         }
         else
         {
             xRotationOffset = 0f;
         }
 
         //check that the pointOfFocus input is strong enough to warrant cam movement on xRotationOffset
         if(!InputManager.rsZAxisInput.IsBetween(-minRSSensitivity, minRSSensitivity))
         {
             zRotationOffset = InputManager.rsZAxisInput * rotationSpeed;
         }
         else
         {
             zRotationOffset = 0f;
         }
     }
 
     void Update()
     {                
         transform.RotateAround(pointOfFocus.transform.position, Vector3.up, xRotationOffset);
         transform.RotateAround(pointOfFocus.transform.position, Vector3.right, zRotationOffset);
         transform.LookAt(pointOfFocus.transform.position);
     }
 }

To use it requires my InputManager as well. The Right Stick input has to be configured to my input manager for it to work:

 public class InputManager : MonoBehaviour {
 
     public static float rsXAxisInput{get; private set;}
     public static float rsZAxisInput{get; private set;}
 
     public static float lsXAxisInput{get; private set;}
     public static float lsZAxisInput{get; private set;}
 
     public delegate void InputControl();
     public static event InputControl characterMove;
     public static event InputControl cameraMove;
     public static event InputControl jump;
     public static event InputControl interact;
 
     void Update()
     {
         lsXAxisInput = Input.GetAxis("Horizontal");
         lsZAxisInput = Input.GetAxis("Vertical");
 
         rsXAxisInput = Input.GetAxis("360_rsHoriz");
         rsZAxisInput = Input.GetAxis("360_rsVert");
 
         //Joystick inputs
         if(Input.GetAxis("Horizontal") != 0 || Input.GetAxis("Vertical") != 0)
         {
             if(characterMove != null){characterMove();}
         }
 
         if(Input.GetAxis("360_rsHoriz") != 0 || Input.GetAxis("360_rsVert") != 0)
         {
             if(cameraMove != null){cameraMove();}
         }
     
         if(Input.GetButtonDown("360_AButton"))
         {
             if(jump != null){jump();}
         }
         if(Input.GetButtonDown("360_XButton"))
         {
             if(interact != null){interact();}
         }
     }
 }

Any suggestions would be appreciated.