Trouble with an Ocarina of Time-styled camera

I've challenged myself with coming up with a camera that should act akin to the camera from Zelda: Ocarina of Time. I've gotten basic movement, more or less, but now I've hit a wall and I can't seem to find a good answer after searching and trying different methods for hours. The movement itself works how I want it to except my camera keeps jittering. Here's my current code:

 using System.Collections;
 using System.Collections.Generic;
 using UnityEngine;
 
 public class CamBehavior : MonoBehaviour
 {
     [SerializeField]
     private GameObject player;              // Reference to player object
     [SerializeField]
     private float heightModifier = 1.5f;    // Height offset of camera relative to player
     [SerializeField]
     private float maxDistance = 10.0f;      // Max distance between player/camera before camera moves towards player
     [SerializeField]
     private float minDistance = 7.0f;       // Minimum distance between player/camera before camera moves away from player
 
     private float camDistance;              // Distance between player and camera
     private PlayerMovement playerMoveScript;    // Reference to the movement script of player
     private Vector3 targetPosition;         // Next target for camera to move towards
 
     public float GetCamDistance
     {
         get
         {
             return camDistance;
         }
     }
 
     void Awake()
     {
         player = GameObject.FindGameObjectWithTag("Player");
         if (player == null)
         {
             Debug.LogError("ERROR: Player GameObject variable not defined.");
         }
 
         playerMoveScript = player.GetComponent<PlayerMovement>();
         transform.position = new Vector3(player.transform.position.x, player.transform.position.y + heightModifier, player.transform.position.z - maxDistance); // Set initial camera position
     }
 
     void LateUpdate()
     {
         targetPosition = new Vector3(player.transform.position.x, player.transform.position.y + heightModifier, player.transform.position.z);
         camDistance = Vector3.Distance(player.transform.position, transform.position);
 
         // Camera movement
         if (playerMoveScript.IsMoving && camDistance >= maxDistance)
         {
             transform.position = Vector3.MoveTowards(transform.position, targetPosition, playerMoveScript.Speed * Time.deltaTime);
         }
         else if (playerMoveScript.IsMoving && camDistance <= minDistance)
         {
             transform.position = Vector3.MoveTowards(transform.position, targetPosition, -playerMoveScript.Speed * Time.deltaTime);
         }
         transform.LookAt(player.transform);
     }
 }
 

After some testing, I figured it could be one of two things (or a combination of the two): Either it's the transform.LookAt that's causing the jitter somehow, or the if statements because the player might be constantly moving in and out of the limits I've set. Any and all help would be highly appreciated because I've been scratching my head at how to solve this logic puzzle for days now.