Return 3rd Person camera to original position

Im trying to build a third person controller. I have a system that uses Raycasting to tell when the camera is going to clip a wall or other object, and moves the camera closer to the player accordingly. However, when a wall or object no longer blocks the camera, the camera remains in the shortened position. I tried to fix it, but now it just jumps back and forth between the standard position and the shortened position every frame, which is much worse.

     x += Input.GetAxis("Mouse X") * xSensitivity * distance;
     y -= Input.GetAxis("Mouse Y") * ySensitivity;

     y = ClampAngle(y, yMinLimit, yMaxLimit);
 
     Quaternion rotation = Quaternion.Euler(y, x, 0);

     if (Physics.Linecast(target.position, transform.position, out RaycastHit hit))
     {
         distance -= hit.distance;
     }
     else
     {
         distance = florp; //this is how i tried to fix it, but it does NOT work
     }

     Vector3 negDistance = new Vector3(0.0f, 0.0f, -distance);
     Vector3 position = rotation * negDistance + target.position;

     transform.rotation = rotation;
     transform.position = position;

Florp is a place holder name. its a float with value 5f. Any and all help is appreciated.