Third Person Top-Down Camera with Faux Gravity

Dear community!

I'm struggling while making a faux gravity based game (planetary gravity). I've got my character controller set up and while testing I have used my main camera as a child object of my player object. This, however, creates stuttering and jittering while moving. I want to move my camera now through scripting so I can maybe try lerp and not have this jitter anymore. I have the following code in my camera script which correctly follows the player but when I want to rotate with the player as well the camera moves all weirdly.

 using UnityEngine;
 using System.Collections;
  
 public class CameraController : MonoBehaviour {
  
     public GameObject player;       //Public variable to store a reference to the player game object
  
  
     private Vector3 offset;         //Private variable to store the offset distance between the player and camera
  
     // Use this for initialization
     void Start ()
     {
         //Calculate and store the offset value by getting the distance between the player's position and camera's position.
         offset = transform.position - player.transform.position;
     }
  
     // LateUpdate is called after Update each frame
     void LateUpdate ()
     {
         // Set the position of the camera's transform to be the same as the player's, but offset by the calculated offset distance.
         transform.position = player.transform.position + offset;
  
  
     }
 }

Here you can find a video of my issue: https://www.youtube.com/watch?v=aDe8stVTpDw

I have found a partial solution by using the RotateAround method on the camera's transform but this gives a slight offset which I think are the cause of rounding errors in my calculations. It also doesn't work when I add in a jumping method for my character.