Beginner orbit camera + player rotation issues

I have two goals with this and I can only get parts of them to work at once.

1) Mouse movement causes the camera to orbit around the player character, as is common in third person games.
2) The player rotates following the camera, so that you can walk by holding W and moving the mouse. Ideally I want the rotation to happen only while you move, and gradually, but that's a later goal.

For now the goal is to simply align the player forward axis with the camera's, without jitter.

Camera script:

 public class CameraFollower : MonoBehaviour
 {
     public GameObject cameraFollowTarget;
 
     private float inputSensitivity = 1.2f;
     private float cameraDistance = 8f;
 
     public float cameraYaw = 0f;
     private float cameraPitch = 20f;
 
     public float minCameraPitch = -60f;
     public float maxCameraPitch = 80f;
 
     private Vector3 cameraOffset = new Vector3();
 
     void Start()
     {
         Quaternion lookRotation = Quaternion.Euler(cameraPitch, cameraYaw, 0f);
         transform.localRotation = lookRotation;
     }
     private void updateCameraAngles()
     {
         float mouseX = Input.GetAxis("Mouse X") * inputSensitivity;
         float mouseY = Input.GetAxis("Mouse Y") * inputSensitivity;
         cameraYaw += mouseX;
         cameraPitch -= mouseY;
         cameraPitch = Mathf.Clamp(cameraPitch, minCameraPitch, maxCameraPitch);
     }
 
     private void LateUpdate()
     {
         updateCameraAngles();
 
         if (Input.GetKey(KeyCode.LeftShift)) {
             // This is really jittery/choppy, but player rotation works great
             Vector3 lookPosition = cameraFollowTarget.transform.position - transform.forward * cameraDistance;
             Quaternion lookRotation = Quaternion.Euler(cameraPitch, cameraYaw, 0f);
             transform.SetPositionAndRotation(lookPosition, lookRotation);
         }
         else
         {
             // This is really smooth, but the player only rotates half the angle I expect
             // (and cameraYaw reflects this half angle, vs the camera's actual rotation)
             cameraOffset.Set(0, 0, -cameraDistance);
             cameraOffset = Quaternion.AngleAxis(cameraPitch, Vector3.right) * cameraOffset;
             cameraOffset = Quaternion.AngleAxis(cameraYaw, Vector3.up) * cameraOffset;
 
             transform.position = cameraFollowTarget.transform.TransformPoint(cameraOffset);
             transform.LookAt(cameraFollowTarget.transform.position);
         }
     }
 }

Player script:

 public class Player : MonoBehaviour
 {
     [SerializeField] private CameraFollower cameraFollower;
 
     private void FixedUpdate()
     {
         // Attempt 1) Shift held: works, but jittery. Shift not held: player rotates HALF the camera angle, but it otherwise works
         // Printing cameraYaw also shows half the angle, e.g. rotate the camera 90 degrees and cameraYaw is 45 (and the character rotates 45 degrees as well)
         transform.rotation = Quaternion.AngleAxis(cameraFollower.cameraYaw, Vector3.up);
 
         // Attempt 2) Shift held: works, but jittery. Shift not held: rotates faster and faster based on mouse position
         // transform.rotation = Quaternion.AngleAxis(cameraFollower.transform.rotation.eulerAngles.y, Vector3.up);
 
         // Attempt 3) Same issues as the one above
         /*
         var rot = cameraFollower.transform.rotation;
         rot.x = 0;
         rot.z = 0;
         transform.rotation = rot;
         */
     }
 }

I can't figure this one out. Why is cameraYaw half the camera rotation angle (the camera's transform.rotation.eulerAngles.y) when shift is not held?

As far as I can tell, the camera tracking isn't dependent on the player rotation, only position, and cameraYaw is only modified by mouse input, so how can it depend on the player rotation?