My character "shakes" when I smooth out the camera movement

Hello, I have a simple character controller and a simple camera controller that is supposed to follow the player in a smooth way (with a lerp). But when I enable smoothing on the camera, the player's movement seems jerky... when I remove the smoothing or when I freeze the camera, the character's movement is nice again !

Here is a video that shows that...

Here's my code :

Character Controller :

 private void Update()
     {
         // (other stuff)

         rb.MovePosition(rigidbody.position + (moveInput * speed * Time.deltaTime));
     }

Camera Controller

     [SerializeField] private Transform player;
     [SerializeField] private float smooth, smoothAngle;
     [SerializeField] private Vector3 offset;
     private Vector3 referenceVelocity = Vector3.zero;
 
     private void Update()
     {
         Vector3 desiredPosition = player.position + offset;
         transform.position = Vector3.SmoothDamp(transform.position, desiredPosition, ref referenceVelocity, smooth * Time.deltaTime);
 
         float smoothedAngle = Mathf.LerpAngle(transform.rotation.eulerAngles.x, angle, smoothAngle * Time.deltaTime);
         transform.rotation = Quaternion.Euler(smoothedAngle, transform.rotation.y, transform.rotation.z);
     }