IsPointerOverGameObject() is true for wrong object sometimes

I have some code for a tooltip UI object that I want trigger after about a half a second. It's working except that I have a bug, caused by IsPointerOverGameObject(), where sometimes the tooltip will pop up over the wrong UI object. This only happens if you move the cursor from one object to the next at a specific speed; otherwise it works perfectly. But it happens way more often than I would like.

The code looks like this:

     public void OnPointerEnter(PointerEventData eventData)
     {
         StartCoroutine(WaitToPop());
         Debug.Log(name+ " OnPointerEnter");
 
     }
 
     public void OnPointerExit(PointerEventData eventData)
     {
         Debug.Log(name + " OnPointerExit");
         toolTipPanel.SetActive(false);
     }
 
     IEnumerator WaitToPop()
     {
         yield return waitTime;
         if (EventSystem.current.IsPointerOverGameObject())
         {
             Debug.Log(name + " IsPointerOverGameObject()");
 
             panelText.text = tooltipText;
             toolTipPanel.transform.position = tooltipPosition;
             toolTipPanel.SetActive(true);
 
         }
     }

The console looks like this:

And here is a gif of the offending code in action:

If necessary, I can upload a simple version of the project somewhere.