Camera Stutter During Distance Correction

My problem is that I'm trying to control the distance between the camera, and target on the z-axis, but the camera stutters while the correction is taking effect. I'm certain that it's because the methods I'm using to push and pull the camera are overshooting, or somehow conflicting with one another.

I'll explain how it works at the bottom:

 using UnityEngine;
 
 public class CameraController : MonoBehaviour
 {
     
     
     public GameObject targetToLookAt;
     Vector3 targetPosition = Vector3.zero;
 
     Vector3 DestinationSphere = Vector3.zero;
 
     public float maxDistance = -5f, minDistance = -1f;
     public float zDistance = -5f;
     
 
     public float yOffset = 0.0f;
     public float MAX_X = 80f, MIN_X = -60f;
     float xAngle, yAngle;
     public float rotationSpeed = 250f;
     Quaternion angleToRotateTowards = Quaternion.identity;
 
 
 
     float timeElapsed;
     public float lerpSpeed = 7f;
     float startValue = 0;
     float endValue = 10;
 
 
 
     Vector3 zPosition = Vector3.zero;
 
     
     RaycastHit hit;
 
     
     // Start is called before the first frame update
     void Start()
     {
         
     }
 
     // Update is called once per frame
     void Update()
     {
         xAngle += Input.GetAxis("Mouse Y") * rotationSpeed * Time.deltaTime;
         yAngle += Input.GetAxis("Mouse X") * rotationSpeed * Time.deltaTime;
         xAngle = Mathf.Clamp(xAngle,MIN_X,MAX_X);
         angleToRotateTowards = Quaternion.Euler(-xAngle,yAngle,0);
 
 
               //Debug.DrawLine(targetToLookAt.transform.position + new Vector3(0, yOffset, 0), transform.position - new Vector3(0, yOffset, 0), Color.red);
         if (Physics.Linecast(targetToLookAt.transform.position + new Vector3(0, yOffset, 0), targetPosition - new Vector3(0, yOffset, 0), out hit))
         {
 
             DestinationSphere = hit.point;
             //Debug.Log(Vector3.Distance(hit.point + new Vector3(0, yOffset, 0), (targetPosition - hit.point) - new Vector3(0, yOffset, 0)));
             //Debug.Log("!!! HIT !!!");
               //Debug.DrawRay(targetToLookAt.transform.position + new Vector3(0, yOffset, 0), (transform.position - targetToLookAt.transform.position) - new Vector3(0, yOffset, 0), Color.black);
             //Debug.DrawRay(hit.point + new Vector3(0, yOffset, 0), (targetPosition - hit.point) - new Vector3(0, yOffset, 0), Color.white);
             
             
 
             if (Vector3.Distance(targetPosition, hit.point) > 0f && Vector3.Dot((targetPosition - hit.point).normalized,(targetToLookAt.transform.position - targetPosition).normalized) == -1) {
 
 
                 //zDistance += lerpSpeed * Time.deltaTime;
                 //zDistance = Mathf.MoveTowards(zDistance, hit.point, lerpSpeed * Time.deltaTime);
                 startValue = zDistance;
                 
                 endValue = -Vector3.Distance(targetPosition, hit.point);
 
                 if (timeElapsed < lerpSpeed)
                 {
 
                     zDistance = Mathf.Lerp(startValue, endValue, timeElapsed / lerpSpeed);
                     timeElapsed += Time.deltaTime;
                     //Debug.Log(startValue);
                 }
                 else
                 {
                     zDistance = endValue;
                 }
 
 
 
             }
             
 
         }
         else if(zDistance > maxDistance)
         {
             //Debug.Log("*** HIT ***");
             //zDistance -= lerpSpeed * Time.deltaTime;
 
             //zDistance = Mathf.MoveTowards(zDistance, maxDistance, lerpSpeed * Time.deltaTime);
 
             startValue = zDistance;
             endValue = maxDistance;
 
             if (timeElapsed < lerpSpeed)
             {
 
                 zDistance = Mathf.Lerp(startValue, endValue, timeElapsed / lerpSpeed);
                 timeElapsed += Time.deltaTime;
             }
             else
             {
                 zDistance = endValue;
             }
 
 
 
 
         }
         //temp += Input.GetAxis("Mouse ScrollWheel") * 50f * Time.deltaTime;
 
 
         zPosition = angleToRotateTowards * new Vector3(0, 0, zDistance);
 
         targetPosition = targetToLookAt.transform.position + zPosition;
         
     }
 
     void LateUpdate() {
         transform.position = targetPosition;
         transform.LookAt(targetToLookAt.transform.position + new Vector3(0,yOffset,0));
     }
 
     private void OnDrawGizmos()
     {
         //Gizmos.color = Color.green;
         //Gizmos.DrawRay(targetToLookAt.transform.position + new Vector3(0, yOffset, 0), (targetPosition - targetToLookAt.transform.position) - new Vector3(0,yOffset,0));
 
 
         //Gizmos.color = Color.cyan;
         //Gizmos.DrawSphere(targetPosition, .3f);
 
         //Gizmos.color = Color.green;
         //Gizmos.DrawSphere(DestinationSphere,.3f);
 
     }
 }
 

targetToLookAt - camera's target (in my project it's just a cube as an example) targetPosition - camera's position. This is meant to hold calculations, and the assigned to the camera's transform.position in LateUpdate().

DestinationSphere - Used in OnDrawGizmos to show the point that the camera must travel to when behind a wall or other geometry.

maxDistance - the maximum distance the camera is allowed to be from the target.

zDistance - the current distance from the target. This is what I'm trying to control smoothly This float variable is used to create a new Vector with the value of zDistance and the rotated via angleToRotateTowards * new Vector(0,0,zDistance).

MAX_X & MIN_X - the clamped angles that the camera is allowed to rotate around the target. angleToRotateTowards - a quaternion that holds the values provided via mouse input.

In the Update I'm first gathering mouse input to rotate the camera around the target and then assigning those values to angleToRotateTowards as a Quaternion.

Before we get into the line casting part, let's finish the rest.

In order to set the camera's position , the quaternion created earlier is multiplied by a new vector that is only defined by the zDistance variable. the multiplication of a quaternion on the left side of the vector Vector(0,0,zDistance) causes it to be rotated by the values of the quaternion. This is all assigned to zPosition.

the targetPosition variable is the set to the target position and then modified by adding the zPosition Vector thereby effectively rotating and pulling the targetPosition away from the target by the zDistance (5 units by default).

This is then set to the camera's transform in LateUpdate(). None of this account for what direction the camera is facing. The transform.LookAt() function takes care of that.

The problem is the stuttering caused by correcting the zDistance. Everytime the camera goes behind any geometry, it's distance to the target must be shortened, and then sent back if there's space behind it. What I'm doing is casting a line(not ray casting) between the camera and target. If the line is intersected by any geometry, the code is meant to "pull" the camera toward the position defined by hit.point.

It works, but the problem comes when adding the push feature. When the line is not being intersected, the code is meant to "push" the camera away back to the maximum distanced allowed (5 by default) I'm thinking that this is the problem . The two conditions are causing a sort of "Tug-Of-War" between pushing and pulling the camera in front of any geometry but also back to it's predefined distance. I've tried lerping, incrementing, and using the MoveTowards function with different result, but the stutter persists. If there's anyone that can help, it would be much appreciated