Issue iterating through gameObjects deleting and addding some

I am looping through up to either the number of slots we have (invSection.childCount) or the number we should have (inv.slotCount).

If a slot is there that shouldn't be, we delete it, and if it isn't there but we need it, we instantiate one and place it in the correct position.

The problem is that removing and adding slots messes up the order and we end up skipping deleting some (or skipping adding some if I use a reverse for-loop).

Oh, and this runs in the editor, which is why I'm using DestroyImmediate() instead if Destroy().

 var invSection = inv.currentInventory.Find("Inventory");
 //This is the parent gameObject to the slots
 
 
 for (int i = 0; i < Mathf.Max(invSection.childCount, inv.slotCount); i++) {
     
     RectTransform slot = null;
     if (i < invSection.childCount) {
         slot = invSection.GetChild(i).GetComponent<RectTransform>();
     }
     
     if (slot != null) {
         if (i < inv.slotCount) {
             if (slot.name != "Slot "+i) { slot.name = "Slot "+i; }
         }
         else {//slot isn't null but is out of range
             DestroyImmediate(slot.gameObject);
         }
     }
     else {
         if (i < inv.slotCount) {//slot is null but within range
             slot = Instantiate(inv.slotPrefab, invSection, false).GetComponent<RectTransform>();
             slot.name = "Slot "+i;
             slot.SetSiblingIndex(i);
         }
     }
 }