How do I associate my custom object for the serializedProperty objectReferenceValue

I'm attempting to write a custom class object that will allow for storing particular data for my editor script functionality.

I have a custom class that I'm using that looks like this:

 [System.Serializable]
 public class ColorList : ScriptableObject
 {
     public List<Color> colors;
     public List<string> labels;
 
     //public Color curColor;
     //public string curLabel;
 
     public ColorList()
     {
       colors = new List<Color>();
       labels = new List<string>();
     }
 
     public void AddColor(Color c)
     {
       colors.Add(c);
     }
 
     public void AddLabel(string s)
     {
       labels.Add(s);
     }
 
     public void At(ref object obj, int index, int list)
     {
       int count = 0;
       // 0 : represents the List<Color>
       // 1 : represents the List<string>
       if (list == 0)
       {
         foreach (Color c in colors)
         {
           if (count == index)
           {
             obj = c;
             return;
           }
 
           ++count;
         }
       }
       else if (list == 1)
       {
         foreach (string s in labels)
         {
           if (count == index)
           {
             obj = s;
             return;
           }
 
           ++count;
         }
       }
     }
 }

Within my Editor functionality, I'm trying to create my reference during the creation of one of the colorAssociations:

 ++colorAssociations.arraySize;
 colorAssociations.GetArrayElementAtIndex(colorAssociations.arraySize - 1).objectReferenceValue = ScriptableObject.CreateInstance<GeometryImageGenerator.ColorList>();
 colorAssociations.serializedObject.ApplyModifiedProperties();

For some reason creating it like this doesn't seem to be setting the objectReferenceValue correctly, it remains as null.

Later on I'm attempt to utilize my object for manipulation, currently this is just test code to get things working:

 GeometryImageGenerator.ColorList colorList = colorAssociations.GetArrayElementAtIndex(i).objectReferenceValue as GeometryImageGenerator.ColorList;
 for (int j = 0; j < colorList.colors.Count; ++j)
 {
    object c = null;
    colorList.At(ref c, j, 0);
    c = EditorGUILayout.ColorField((Color)c);
 }

However, colorList remains to be null. What am I doing wrong and how should I be approaching this?