Rect Transform moving to Wrong Spot

I have the following

             public void GetInventory()
 {
     highlightedInventorySlot = 0;
     OpenUIPanel(0);
     for (int i = 0; i < PlayerInfoManager.instance.inventorySlotCount; i++)
     {
         GameObject slot = Instantiate(inventorySlot, transform.position, Quaternion.identity);
         slot.transform.SetParent(inventoryPanel);
         slot.transform.localScale = Vector3.one;
         inventorySlots.Add(slot);
         InventorySlot iSlot = slot.GetComponent<InventorySlot>();
         iSlotList.Add(iSlot);
         iSlotList[i].slotNum = i;

         if (InventoryManager.instance.inventoryItemList.Count > i)
         {
             iSlotList[i].farmTool = InventoryManager.instance.inventoryItemList[i];
             iSlotList[i].SetTool();
         }
     }
     RectTransform rect = inventorySlots[highlightedInventorySlot].GetComponent<RectTransform>();
     StartCoroutine(SetArrowPosition(new Vector3(rect.position.x, rect.position.y + rect.sizeDelta.y * 3, rect.position.z), new Vector3(0,0,-90)));
     inventoryPanelArrow.gameObject.SetActive(true);
 }

 

which calls

     public IEnumerator SetArrowPosition(Vector3 position, Vector3 rotation)
     {
         yield return new WaitForEndOfFrame();
         inventoryPanelArrow.transform.position = position;
         inventoryPanelArrow.transform.eulerAngles = rotation;
     }
 

The idea here is that my "Inventory Arrow" (an image that hovers over the current item) should move to the currently highlighted slot. (Which upon opening the inventory will be slot 0). However, upon calling SetArrowPosition in GetInventory() the position is wrong, the arrow is off the screen. However, if I then call SetArrowPosition() again by moving to the next slot, with the following.

     public void NextInventorySlot(InputAction.CallbackContext value)
     {
         Vector2 v = value.ReadValue<Vector2>();
         if (value.started)
         {
             if (v.x < -.1f || v.x > .1f)
             {
                 if (v.x < 0)
                 {
                     if (highlightedInventorySlot > 0)
                     {
                         highlightedInventorySlot--;
                     }
                 }
                 if (v.x > 0)
                 {
                     if (highlightedInventorySlot < inventorySlots.Count - 1)
                     {
                         highlightedInventorySlot++;
                     }
                 }
             }    
             RectTransform rect = inventorySlots[highlightedInventorySlot].GetComponent<RectTransform>();
             StartCoroutine(SetArrowPosition(new Vector3(rect.position.x, rect.position.y + rect.sizeDelta.y * 3, rect.position.z), Vector3.zero));
         }
     }
 

It will now be in the right position.

So my question is, why is the position wrong when first called in GetInventory(), but when called later with NextInventorySlot() it's correct.