ScreenToWorldPoint causing OnPointerEnter/Exit to fire multiple times

Hi, Trying to create a tooltip using Screenspace.camera - so instead of just doing mytransform.transform.position = input.mouseposition I have to do some calculations like so;

 var screenPoint = new Vector3(Input.mousePosition.x , Input.mousePosition.y);
 screenPoint.z = 10.0f; //distance of the plane from the camera
 ToolTipGO.transform.position = Camera.main.ScreenToWorldPoint(screenPoint);

The tooltip is beeing shown and follows the mouse araound perfectly, but theres a problem with multiple fireings of onPointerEnter and onPointerExit, it spams my log, and obviously causes the tooltip-gameobject to be instantiated and deleted, making it look like its flickering.

I have tried switching to diffrent Camera.main.DiffrentTypesOfFunctions (and even disabled that last line) and this fires the events as expected - 1 time, solving all issues appart from the tooltip not beeing at the mouse position...

My scripts are as follows:

 public class Tooltip : MonoBehaviour, IPointerEnterHandler, IPointerExitHandler
 {
         GameObject UIManager;
         public string _thisTooltipsString;

         void Start()
         {
             UIManager = GameObject.Find("UIManager");
         }
     
     public void OnPointerEnter(PointerEventData eventData)
         {
            
                 UIManager.GetComponent<TooltipMouseController>().CreateToolTip(thisTooltipsString);
         }
     
         public void OnPointerExit(PointerEventData eventData)
         {
             print("Fire");
             UIManager.GetComponent<TooltipMouseController>().DestroyToolTip();
         }
 }

This script is attached to every gameobject i want to contain a tooltip

Next i have this attached to my UImanager object:

     public class TooltipMouseController : MonoBehaviour
     {
         GameObject ToolTipPrefab;
         GameObject ToolTipGO;
     
         void Start()
         {
             ToolTipPrefab = Resources.Load("ToolTipPrefab") as GameObject;
         }
     
         public void CreateToolTip(string s)
         {
                 ToolTipGO = Instantiate(ToolTipPrefab);
                 ToolTipGO.transform.SetParent(GameObject.Find("Background").transform , false);
                 ToolTipGO.transform.FindChild("Text").GetComponent<Text>().text = s;
         }
     
         public void DestroyToolTip()
         {
             Destroy(ToolTipGO.gameObject);
         }
     
         void LateUpdate()
         {
             if( ToolTipGO != null )
             {
                 var screenPoint = new Vector3(Input.mousePosition.x , Input.mousePosition.y);
                 screenPoint.z = 10.0f; //distance of the plane from the camera
                 ToolTipGO.transform.position =  Camera.main.ScreenToWorldPoint(screenPoint); //This last line can be commented out, and onPointerEnter/Exit only fires once like its supposed to - but tooltip position is ofc no longer in the correct position
             }
         }
     }

So am i doing something wrong? Why does it keep fireing on LateUpdate calculations?