Why is my game lagging when I use Unity's animator to change color of a image in Unity3D?

I'm using Unity's animator to change the color of two images as well as a text color. The color change is on a loop until the player presses "start" and then it is disabled at which point the game runs smoothly. There is some automated character movement going on in the background that starts to lag big time when the images/text are changing color.

Ive also tried lerping the colors in code and experienced the same lag. Do image color changes typically slow down games like this? Any ideas on how to fix this?

Here are two pics of my setup: