UI element highlight OnPointerEnter effected by Hierarchy order of children?

I have made a simple inventory system that lets you drag and drop items around in different spots. I'm using the OnDrag, OnEndDrag, OnPointerEnter, and OnPointerExit events for most the code.

I am having this weird issue where I try to drag and drop/move around a selected item, the mouse/pointer will only trigger OnPointerEnter for inventory spots that appear after the original inventory spot clicked in the game Hierarchy system. e.g. dragging an item from inventory slot #6 (child object #6) will only allow my mouse pointer to enter inventory slots from #6+ and won't trigger on slots #1-#5.

This is illustrated in the gif below where my inventory slots (buttons) change to their highlighted button colors when the mouse hovers over them. This seems to only work for slots (buttons) that appear after the original slot clicked in the game Hierarchy.

Is there some kind of canvas/children ordering of OnPointerEnter events when tied with OnDrag events?

Gif demo of problem

If it helps, here is the simple code attached to the various items. The Inventory system is a set of UI button gameobjects with each having a UI Image gameobject attached as a child.

Attached to every UI image gameobject within each inventory slot:

 public class ItemDragHandler : MonoBehaviour, IDragHandler, IEndDragHandler
 {
     public void OnDrag(PointerEventData eventData)
     {
         // drag UI image object around the screen
         transform.position = Input.mousePosition;
     }
 
     public void OnEndDrag(PointerEventData eventData)
     {
         // ask the inventory manager to decide where this item will end up going
         transform.parent.parent.GetComponent<InventoryManager>().ItemDropEvent(transform.parent.GetComponent<InventorySlotHoverHandler>().InvSlotId, transform);        
     }
 }

Attached to each inventory slot (UI button object):

 public class InventorySlotHoverHandler : MonoBehaviour, IPointerEnterHandler, IPointerExitHandler
 {
     int invSlotId;
     public int InvSlotId
     {
         get { return invSlotId; }
     }
 
     void Start()
     {
         // assign inventory slot ID number based on gameObject name
         invSlotId = int.Parse(Regex.Replace(gameObject.name, "[^0-9]", ""));
     }
     public void OnPointerEnter(PointerEventData eventData)
     {
         // tell the inventory manager that the mouse is hovering over this slot
         transform.parent.gameObject.GetComponent<InventoryManager>().UpdateInventoryHoverStatus(invSlotId, 1);
     }
 
     public void OnPointerExit(PointerEventData eventData)
     {
         // tell the inventory manager that the mouse is no longer hovering over this slot
         transform.parent.gameObject.GetComponent<InventoryManager>().UpdateInventoryHoverStatus(invSlotId, 0);
     }
 }

I have an inventory manager script that just keeps track of which slots are occupied or not as well as which slots currently have the mouse hovering over them.