Dynamically Added Listener to UI Button Not Calling With Variable Parameters

I'm making a role-based avatar selector. First the player selects their role and then picks from a set of avatars dressed like that role. Here are some subclasses that I've made to construct the avatar sets:

 [Serializable]
 public class Avatar {
     public string name;
     public Sprite icon;
 }
 
 [Serializable]
 public class RoleAvatarSet {
     public Toggle avatarPanelToggle;
     public RectTransform avatarHolder;
     public Avatar[] avatars;
 }

 public RoleAvatarSet[] roleAvatarSets;

I've filled 4 roleAvatarSets in the inspector with a variable number of avatars in each (from 1 to as many as 8). Next I needed a way to dynamically load those into my UI:

 void Start() {
     LoadAvatars();
 }
 
 void LoadAvatars() {
     for(int i=0;i<roleAvatarSets.Length;i++) {
         for(int j=0;j<roleAvatarSets[i].avatars.Length;j++) {
             GameObject avatarSelectionButton = (GameObject)Instantiate(avatarPrefab);
             avatarSelectionButton.transform.SetParent(roleAvatarSets[i].avatarHolder, false);
             avatarSelectionButton.transform.FindChild("Image").GetComponent<Image>().sprite = roleAvatarSets[i].avatars[j].icon;
             avatarSelectionButton.GetComponent<Button>().onClick.AddListener(() => SelectAvatar(i, j));
         }
         roleAvatarSets[i].avatarPanelToggle.GetComponentInChildren<Image>().sprite = roleAvatarSets[i].avatars[0].icon;
     }
 }
 
 public void SelectAvatar(int roleSet, int avatar) {
     Debug.Log(roleSet + " : " + avatar);
     selectedAvatar = roleAvatarSets[roleSet].avatars[avatar];
     roleAvatarSets[roleSet].avatarPanelToggle.GetComponentInChildren<Image>().sprite = roleAvatarSets[roleSet].avatars[avatar].icon;
 }

And it all loads up nicely. The problem is when I click one of the dynamically added avatar selection buttons, I get an IndexOutOfRangeException. When I debug what indexes the button is calling with, this is no surprise. Every one - no matter which button I click - calls SelectAvatar with a 4 for 'roleSet' and a 5 for 'avatar'. I only have 4 role sets, so 3 should be my highest index. And I have no clue why avatar is a 5.

I've also debugged my LoadAvatars method and everything checks out with the for loop indexes. It's like something gets messed up with AddListener. Am I using it incorrectly? Am I missing something? Any help would be appreciated.

Thanks!