Camera gets closer to the player when looking at it from certain angles.

So i have this game where you roll a ball, and a camera follows that ball and the ball moves forward in whatever direction the camera is facing it. My problem is that when ever i look at it from the sides or above or below it the camera starts to zoom in and and then zooms out when i get to an angle it likes.

Heres my code for the camera:

      using UnityEngine;
      using System.Collections;
      
      public class MoveCamera : MonoBehaviour {
      
          public float turnSpeed = 4.0f;
          public Transform player;
      
          public float height = 1f;
          public float distance = 2f;
          
          private Vector3 offsetX;
          private Vector3 offsetY;
          
          void Start () {
      
              offsetX = new Vector3 (0, height, distance);
              offsetY = new Vector3 (0, 0, distance);
          }
          
          void LateUpdate()
          {
              offsetX = Quaternion.AngleAxis (Input.GetAxis("Mouse X") * turnSpeed, Vector3.up) * offsetX;
              offsetY = Quaternion.AngleAxis (Input.GetAxis("Mouse Y") * turnSpeed, Vector3.right) * offsetY;
              
              
              transform.position = player.position + offsetX + offsetY; 
              transform.LookAt(player.position);
          }
      }
      

I dont know what to do. Im new to coding in c# and have already tried everything i can think of.