Why Do I Have Camera Stutter?

I am having a problem where the camera seems extremely stuttery. Here are my scripts:

 public class PlayerFollower : MonoBehaviour {
     [Tooltip("Player GameObject")]
     public GameObject player;
     private Vector3 currentNormal;
     private Vector3 cameraRotation;
     private Quaternion toRotation;
     private Quaternion initialRotation;
     // Use this for initialization
     void Start () {
         currentNormal = Vector3.up;
         toRotation = transform.rotation;
     }
 
     private void Update()
     {
         Vector3 newNormal = player.GetComponent<PlayerController>().getNormal();
         if (!newNormal.Equals(currentNormal))
         {
             cameraRotation = Vector3.Cross(newNormal.normalized, currentNormal.normalized).normalized;
             toRotation *= Quaternion.FromToRotation(currentNormal.normalized, newNormal.normalized);
             currentNormal = newNormal;
         }
     }
 
     void LateUpdate () {
 
         //Follow Player
         transform.position = player.transform.position;
         //Rotate Normal Vector as Necessary
         transform.rotation = Quaternion.Slerp(transform.rotation, toRotation, Time.deltaTime * 2.0f);
     }
 }

  

 public class CameraController : MonoBehaviour {
     [Tooltip("Follower GameObject")]
     public GameObject follower;
 
     void LateUpdate () {
         
         //Rotate for Player Controlled Rotation
         transform.RotateAround(follower.transform.position, follower.transform.up, Input.GetAxis("Look X") * 300.0f*Time.deltaTime);
     }
 }

So essentially, the camera is a child to a game object that follows the player, and rotates based on the degree of the incline the player is on. The camera rotates around the follower based on the horizontal mouse position/right joystick horizontal axis.

The stuttering is pretty noticeable to me and I'm not sure what's wrong with what I'm doing. I initially thought that it might have something to do with the interaction of the two separate rotations I'm doing, but I commented out the one in PlayerFollower and continued to experience stutter. Please help!

P.S. For mouse and keyboard, "Look X" is equivalent to Unity's default "Mouse X". For the gamepad, "Look X" is the horizontal axis of the right thumbstick.