PointerEnter / PointerExit keeps triggering on UI Element

Hey guys,

I've been trying to create a Point & Click game in Unity and I'm making a type of tooltip/item name showing when hovering the mouse over an Item in the Scene and in the UI Inventory.

The first part of it works as I coded this in the Item. The player can mouseOver the object and it displays the name in a textbox. The second part not so much, as I programmed it in the Inventory slot code with eventSystems the same way as the first part works.

Code for the first part

 public void OnMouseEnter(){
     if (textObject == null) {
         textObject = Instantiate (gameManager.TextBoxPrefab);
         textObject.name = "Item - " + itemName + " - TextBox";
         textObject.transform.SetParent (GameObject.FindGameObjectWithTag("UI").transform, false);
         textObject.transform.position = Input.mousePosition;
         textObject.GetComponentInChildren<Text> ().text = itemName;
     }
 }
 
 public void OnMouseExit(){
     Destroy (textObject);
 }
 
 // This works well!

Problem | The second part doesn't work as it is done with UI elements. I tried to use the EvenSystems and they do register the PointerEnter and Exit at the inventory slots, But they rapidly turn on and off for some reason (resulting in flickering textBoxes appearing and disappearing).

 public void OnPointerEnter(PointerEventData data){
     if (textObject == null && Occupied) {
             textObject = Instantiate (gameManager.TextBoxPrefab);
             textObject.name = "Item - " + item.itemName + " - TextBox";
             textObject.transform.SetParent (GameObject.FindGameObjectWithTag("UI").transform, false);
             textObject.transform.position = Input.mousePosition;
             textObject.GetComponentInChildren<Text> ().text = item.itemName;
     }
 }
 
 public void OnPointerExit(PointerEventData data){
     Destroy (textObject);
 }

Question: How can I make this work the same way as the mouseEnter and mouseExit events and what causes the flickering or not working PointerEnter and Exit?

Thanks in advance!

EDIT 1 | The inventory slots exist out of a Panel with an Image object parented to it - I did stop raycasting for the Image so that they dont overlap each other.