Camera Follow - SmoothDamp Inconsistency

I've got a script on my main camera that is following the main character. I use Mathf.SmoothDamp to ease to its position. This works fine but the problem is that I want to change the smoothTime variable on the fly. When the main character hits a wall, the game is over or GameManager.gameOver = true. Once the game is "over" I want to change smoothTime from .1f to 1.0f to give a slow ease effect.

Here is my code:

     void Start () {
         smoothTime = .1f;
     }
 
     void Update() {
         if(GameManager.gameOver) {
             smoothTime = 1f;
         }
     }
 
     void LateUpdate () {
         Vector3 tempV3 = new Vector3(target.position.x, target.position.y, transform.position.z);
         transform.position = Vector3.SmoothDamp(transform.position, tempV3, ref tempVel, smoothTime);
     }

Most of the time this works, but it sometimes is clearly easing with a smoothTime of .1f instead of 1. Is it possible that when it checks for GameManager.gameOver it is getting a value of false, but by the time the SmoothDamp has been updated gameOver is actually true?

Another possibility is that the velocity is too low and/or the camera is too close to the target position when the SmoothDamp is updated/fired off... I say this because sometimes (very often, actually) this problem occurs when it takes a small movement with relatively low velocity to hit a wall, end the game and cause the smoothTime ease to be too short.

Is there any reason this shouldn't work as intended? Do I even need the LateUpdate() function in this case?

Thanks in advance,

-Shane