Update function not being called/freezing.

In my game when you get within a certain radius of an object an interaction icon appears. That all works fine except for the interaction. I'm going with a hold down blank button to interact sort of thing. In my world some of the object work I can hold down the interaction button and the script runs perfectly fine, but some are sort of frozen. Ill go up to them and they'll be stuck in the middle and wont accept new input and then at random times for a few seconds they will start working again and allow interaction and then they freeze up again. I added a debug.log to their update function and for whatever reason it seems the update function is not running even though others work perfectly fine and I have confirmed they are on an active gameobject and the script is active and there are no errors in the console. Here is the code:

 using System;
 using System.Collections;
 using System.Collections.Generic;
 using UnityEngine;
 using UnityEngine.Events;
 using UnityEngine.Serialization;
 using UnityEngine.UI;
 
 //A class for making the interact hold circles that appear in the world
 public class holdButtonInteraction : MonoBehaviour
 {
     [Serializable]
     public class ButtonClickedEvent : UnityEvent { }
 
     [FormerlySerializedAs("onClosed")]
     [SerializeField]
     private ButtonClickedEvent m_OnClick = new ButtonClickedEvent();
 
 
     [Tooltip("The key that we should listen for.")]
     public KeyCode keyToListenFor;
 
     [Tooltip("The time you should be required to hold down the key for. [SECONDS]")]
     public float holdTime;
 
     [Tooltip("The image who's fill should be adjusted")]
     public Image iRenderer = null;
 
     //A float to keep track of how long you have held the button/how much to fill the image
     public float time;
 
     // Start is called before the first frame update
     void Start()
     {
         //Make sure the time is reset
         time = 0;
 
         //Make sure the image is valid, if not set it equal to the image attached to this gameobject
         if (!iRenderer) { iRenderer = GetComponent<Image>(); }
     }
 
     // Update is called once per frame
     void Update()
     {
         //Make sure Irenderer is not null (If there is no image object attached to the gameobject)
         if (iRenderer)
         {
             //Update the fill on the image
             Debug.Log("Fill adjusted!"); //Debug.Log that is not being called
             iRenderer.fillAmount = time;
         }
         else
         {
             //Warn in the console
             Debug.LogWarning("No image component attached to GameObject or iRenderer component is set to an inccorect value.");
         }
 
         //Check if we have pressed down the key
         if (Input.GetKey(keyToListenFor))
         {
             //When we are pressing the key increase the value of time
             //Ensure we do not go over 1
             if (time < 1)
             { 
                 //Increment the time by 1 / holdtime each frame use time.deltatime to find out how much to increment by
                 time += 1 / holdTime * Time.deltaTime;
             }
         }
         else
         {
             //Decrement the time var when not pressed
             if (time > 0)
             { time -= 0.1f / holdTime * Time.deltaTime * 3; }
         }
 
         //Check if we have met the time requirement and if so reset the time and invoke the event
         if (time >= 1) 
         {
             time = 0;
             m_OnClick.Invoke();
         }
     }
 
 
 }

BTW: I'm on Unity 2019.4.1f1 and am upgrading to 2019.4.4f1 to see if that changes anything. Also, I have another monobehaviour on the parent object of said script and when one freezes they both do.