How do I get my IEnumerator to readjust scale when time is added/taken away from timer script?

I have an IEnumerator for a coroutine. Basically, what it is doing is making a game object decrease its size over time.

It does this just fine. But the issue I'm having is that when I add or take away time the scale of the object doesn't adjust according to the new time.

The player object script:

 // At the start of the game..
 void Start()
 {
     // Assign the Rigidbody component to our private rb variable
     rb = GetComponent<Rigidbody>();

     // Set the text property of the Win Text UI to an empty string, making the 'You Win' (game over message) blank
     winTextObject.SetActive(false);

     gameOverText.SetActive(false);
     //start my lerp coroutine
     StartCoroutine(LerpFunction(targetScale, timer.timeRemaining));
 }



 void FixedUpdate()
 {
     // Create a Vector3 variable, and assign X and Z to feature the horizontal and vertical float variables above
     Vector3 movement = new Vector3(movementX, 0.0f, movementY);

     rb.AddForce(movement * speed);
 }

 void OnTriggerEnter(Collider other)
 {
     // ..and if the GameObject you intersect has the tag 'Pick Up' assigned to it..
     if (other.gameObject.CompareTag("PickUp"))
     {
         other.gameObject.SetActive(false);

         // add 5 seconds to the timer
         timer.timeRemaining += 5;
     }
     // ..and if the GameObject you intersect has the tag 'Fire' assigned to it..
     if (other.gameObject.CompareTag("Fire"))
     {
         other.gameObject.SetActive(false);

         // take away 5 seconds to the timer
         timer.timeRemaining -= 5;

     }
 }
 //create an IEnumerator for my Lerp
 IEnumerator LerpFunction(float endValue, float duration)
 {
     float time = 0;
     float startValue = scaleModifier;
     Vector3 startScale = transform.localScale;

     while (time < duration)
     {
         scaleModifier = Mathf.Lerp(startValue, endValue, time / duration);
         transform.localScale = startScale * scaleModifier;
         time += Time.deltaTime;
         yield return null;
     }
     transform.localScale = startScale * targetScale;
     scaleModifier = targetScale;
 }
 void OnMove(InputValue value)
 {
     Vector2 v = value.Get<Vector2>();

     movementX = v.x;
     movementY = v.y;
 }
 }

The timer script:

 private void Start()
 {
     // Starts the timer automatically
     timerIsRunning = true;
 }

 void Update()
 {
     if (timerIsRunning)
     {
         if (timeRemaining > 0)
         {
             timeRemaining -= Time.deltaTime;
             DisplayTime(timeRemaining);
         }
         else
         {
             Debug.Log("Time has run out!");
             timeRemaining = 0;
             timerIsRunning = false;
         }
     }
 }

 void DisplayTime(float timeToDisplay)
 {
     timeToDisplay += 1;

     float minutes = Mathf.FloorToInt(timeToDisplay / 60);
     float seconds = Mathf.FloorToInt(timeToDisplay % 60);

     timeText.text = string.Format("{0:00}:{1:00}", minutes, seconds);
 }

}

I'm wondering if the yield return null is what's causing it just to begin the coroutine again according to the original time. If it is, how would I get it to update the new time remaining?