Buttons become invisible when changing color (on Button or on Image) in script

I'm trying to make a UI that can add buttons to a panel and change their colors based on the player's progress in the game. I have a prefab button that I instantiate to create the list of buttons. Here's where I set up the buttons:

 foreach(Quest q in QuestManager.questManager.quests){
             GameObject buttonObj = Instantiate(questButtonPrefab) as GameObject;
             buttonObj.transform.SetParent(gameObject.transform.GetChild(0), false);
 
             RectTransform rt = buttonObj.GetComponent<RectTransform>();
             rt.localPosition = new Vector3(0, buttonY, 0);
             rt.sizeDelta = buttonSize;
             buttonY -= (int)(buttonSize.y);
 
             Button button = buttonObj.GetComponent<Button>();
             button.onClick.AddListener(() => questButtonClicked(q.id));
             button.transition = Selectable.Transition.None;
             questButtons.Add(button);
 
             Text buttonText = button.transform.GetChild(0).gameObject.GetComponent<Text>();
             buttonText.text = q.questName;
 
             Image buttonImage = buttonObj.GetComponent<Image>();
 
             if(q.completed){
                 //button.colors = completedColorBlock;
                 buttonImage.color = completedColor;
             } else if(q.started){
                 //button.colors = inProgressColorBlock;
                 buttonImage.color = inProgressColor;
             } else if(q.isAvailable()){
                 //button.colors = availableColorBlock;
                 buttonImage.color = availableColor;
             } else {
                 //button.colors = unavailableColorBlock;
                 buttonImage.color = unavailableColor;
             }
         }

When I try to change the colors, either on the image or the button components, the buttons disappear, but they still work if you click on where they should be. The Text object (child of the button) still displays. This only happens when I try to change the colors in the script, and works as expected when I change the colors in the editor while the game is running. Can anyone explain what is going on here?