Why are my objects randomly brighter than the sun? Specular? Lighting issue?

Both of these objects have the exact same Hex colors.

I've got my objects set to change colors in random directions in steps of random(+ or -0.1) and the colors are changing their values like I was expecting them to. But for some reason it's randomly making the objects EXTREMELY shiny/specular and I am not really sure what's going on. I have a sneaking suspicion that my issue comes from the fact that I'm using InvokeRepeating() to loop through the colors instead of using the main Update() loop. But I needed to be able to slow the color changing down so I don't have a seizure. If I pause the game and slide the HSV bar it fixes immediately.

     private void Start()
     {
         // Start calling ChangeColor() on a loop every x seconds
         InvokeRepeating("ChangeColor", 0.01f, changeSpeed);
     }
 
 
     void ChangeColor()
     {
         Color oldColor = gameObject.GetComponent<Renderer>().material.color;
         Color newColor = GetNewColor(oldColor);
         gameObject.GetComponent<Renderer>().material.color = newColor;
     }
 
     Color GetNewColor(Color oldColor)
     {
         float H, S, V;
         Color.RGBToHSV(oldColor, out H, out S, out V);
 
         H = ConvertMinMaxColor(Random.Range(H - maxChange, H + maxChange));
         S = ConvertMinMaxColor(Random.Range(S - maxChange, S + maxChange), true);
         V = ConvertMinMaxColor(Random.Range(V - maxChange, V + maxChange));
 
         Color rtnColor = Color.HSVToRGB(H, S, V);
         return rtnColor;
     }
 
     // If number is over 1 it will roll over to 0.
     // If number is under 0 it will roll under to 1
     float ConvertMinMaxColor(float number, bool isSaturation = false)
     {
         if (isSaturation)
         {
             if(number >= 1f)
             {
                 return number - 3f;
             }
             if(number <= 7f)
             {
                 return number + 3f;
             }
         }
         else
         {
             if (number >= 1f)
             {
                 return number - 1f;
             }
             if (number <= 0f)
             {
                 return number + 1f;
             }
         }
         return number;
     }

Any help would be appreciated, thank you!