Problems with EventSystem when implementing a custom raycaster

Context: I'm rendering a 3D scene to a RenderTexture and showing it on a UI RawImage. In order to implement correct mousebehaviour, I decided to make my own implementation of a Raycaster, by rendering a second RenderTexture which uses a replacement shader. This shader gets a color value at object instantiation which corresponds to a unique ID. The ID<>GameObject references are then stored in a dictionary for easy retrieval.

I based this "SelectionTexture" idea on the following article: https://web.archive.org/web/20170828092436/http://blog.celestial-static-games.de/2017/08/25/pixel-perfect-selection-using-shaders/

And decided that it'd be great to have simple eventsystem handlers on my selectable gameobjects (ie IPointerClickHandler etc) that are being rendered on the texture. Realized I could do this for non-UI gameobjects when stumbling upon this: https://answers.unity.com/questions/1161275/can-i-make-a-non-ui-gameobject-draggable-by-implem.html

So I made a CustomRaycaster which inherits from BaseRaycaster and implemented both Raycast() and eventCamera. Inside raycast it'll check if the pointer is over the corresponding UI texture and if yes, it'll do some calculations to get the corresponding pixel's color from the texture. It then uses that color to get the corresponding gameobject from the dictionary. rgba(0,0,0,0) meaning no selection.

So far so good! This is actually working quite well and gives us pixel-perfect selection without the use of any colliders or some type of mesh-raycasting. Furthermore, we can have multiple cameras w their own RenderTextures to show multiple viewpoints of the scene (I'm making a type of editor, as you might have guessed).

Where it goes wrong: I put a cube in the scene, which is being correctly found by my custom raycaster. I then added a monobehaviour with IPointerClickHandler and IPointerEnterHandler. The weird thing is that OnPointerEnter and OnPointerExit are being called EVERY frame that my mouse is over the object. And OnPointerClick, is only being called once every 5-10 clicks. So it kinda works, but not as intended.

So I'm guessing I'm missing some important detail about how the EventSystem interprets the raycastresult. Perhaps the values in my result are incorrect? Here's the code:

 public class CustomRaycaster : BaseRaycaster
     {
 
         public override int sortOrderPriority
         {
             get
             {
                 return base.sortOrderPriority;
             }
         }
 
         public override int renderOrderPriority
         {
             get
             {
                 return base.renderOrderPriority;
             }
         }
 
         protected CustomRaycaster()
         { }
 
 
         private IRenderSelectionTextureContainer m_SceneView;
         /// <summary>
         /// SceneView is the View-behaviour attached to the actual UI-Image which displays the RenderTexture
         /// It holds references to:
         /// - The actual 3D scene with object map
         /// - The dynamic panel util -> panel
         /// - The RenderTexture
         /// - The SelectionTexture -> texture
         /// </summary>
         private IRenderSelectionTextureContainer sceneView
         {
             get
             {
                 if (m_SceneView != null)
                     return m_SceneView;
 
                 m_SceneView = GetComponent<IRenderSelectionTextureContainer>();
                 if (m_SceneView == null)
                     Debug.LogError("Failed to find IRenderSelectTextureContainer as a component");
                 return m_SceneView;
             }
         }
 
         private ISelectionTextureObjectMapContainer m_objMapContainer;
         private ISelectionTextureObjectMapContainer objMapContainer
         {
             get
             {
                 if (m_objMapContainer != null)
                     return m_objMapContainer;
 
                 m_objMapContainer = sceneView.GetObjMapContainer();
                 return m_objMapContainer;
             }
         }
 
         private PanelUtil m_Panel;
         /// <summary>
         /// Easy access to info about the dynamic panel to find
         /// - center
         /// - corners
         /// - width/height, etc
         /// </summary>
         private PanelUtil panel
         {
             get
             {
                 if (m_Panel != null)
                     return m_Panel;
 
                 m_Panel = sceneView.GetPanelUtil();
                 return m_Panel;
             }
         }
 
         private RenderTexture m_Texture;
         /// <summary>
         /// The SceneView's SelectionTexture
         /// </summary>
         private RenderTexture texture
         {
             get
             {
                 if (m_Texture != null)
                     return m_Texture;
 
                 m_Texture = sceneView.GetSelectionTexture();
                 return m_Texture;
             }
         }
 
         private RectTransform m_RectTransform;
         /// <summary>
         /// The SceneView's RectTransform
         /// </summary>
         private RectTransform rectTransform
         {
             get
             {
                 if (m_RectTransform != null)
                     return m_RectTransform;
 
                 m_RectTransform = transform as RectTransform;
                 return m_RectTransform;
             }
         }
 
         // calculation stuff
         Vector2 localPos; //MousePos
         Vector3[] localCorners; //LocalCorners of the transfo
         float imgWidth, imgHeight;
         Vector2 positionNormalizedForTexCoords;
         Texture2D texture2D, tempTex; //Temp "screenShot" of the rendertext
         Color32 selectCol;
 
         // This is the generalized raycast function (called like any other type of raycaster)
         public override void Raycast(PointerEventData eventData, List<RaycastResult> resultAppendList)
         {
             if (texture == null || !sceneView.IsMouseOver)
                 return;
 
             if(texture2D == null || tempTex == null || localCorners == null)
             {
                 localCorners = new Vector3[4];
                 texture2D = new Texture2D(texture.width, texture.height, TextureFormat.ARGB32, false);
                 tempTex = new Texture2D(texture.width, texture.height);
             }
 
             RectTransformUtility.ScreenPointToLocalPointInRectangle(rectTransform, eventData.position, eventCamera, out localPos);
 
             rectTransform.GetLocalCorners(localCorners);
             imgWidth = localCorners[3].x - localCorners[0].x;
             imgHeight = localCorners[1].y - localCorners[0].y;
 
             positionNormalizedForTexCoords.x = localPos.x / imgWidth + 0.5f;
             positionNormalizedForTexCoords.y = localPos.y / imgHeight + 0.5f;
 
             RenderTexture.active = texture;
             texture2D = GetRTPixels(texture);
 
             selectCol = texture2D.GetPixel((int)(positionNormalizedForTexCoords.x * (float)texture.width), (int)(positionNormalizedForTexCoords.y * (float)texture.height));// .GetPixelBilinear(positionNormalizedForTexCoords.x, positionNormalizedForTexCoords.y);
             //selectCol = texture2D.GetPixelBilinear(positionNormalizedForTexCoords.x, positionNormalizedForTexCoords.y);
             RenderTexture.active = null;

             // If color is (0,0,0,0) it means no selection
             if (objMapContainer.ColorToUInt(selectCol) == 0)
                 return;
 
             //Debug.Log("Color is: " + selectCol + " texcoord: " + positionNormalizedForTexCoords + " localpos mouse: "+localPos);
 
             GameObject go = objMapContainer.GetGameObjectByColor(selectCol);
             //Debug.Log("Found object: " + go.name);
 
             if (go == null)
             {
                 Debug.LogWarning("Got a null GO from the map? why?");
                 return;
             }
 
             // Can there ever be more than 1 result?? I don't think so
 
             var castResult = new RaycastResult
             {
                 gameObject = go,
                 module = this,
                 distance = 0,
                 index = resultAppendList.Count,
                 depth = 100
             };
             resultAppendList.Add(castResult);
 
         }
 
         public override Camera eventCamera
         {
             get
             {
                 return sceneView.Cam2D;
             }
         }
 
 
 // TODO- this code is weird, needs to be refactored
         public Texture2D GetRTPixels(RenderTexture rt)
         {
             RenderTexture currentActiveRT = RenderTexture.active;
 
             RenderTexture.active = rt;
 
             // Create a new Texture2D and read the RenderTexture image into it
             //Texture2D tex = new Texture2D(rt.width, rt.height);
 
             tempTex.ReadPixels(new Rect(0, 0, tempTex.width, tempTex.height), 0, 0);
 
             RenderTexture.active = currentActiveRT;
             //  Destroy(tex, );
 
             return tempTex;
         }
 
 
     }