GraphicRaycaster seems to ignore Toggle elements

Hello everybody.

Here's what I have: I have a canvas with a bunch of buttons and a checkbox (Toggle) on it. Please excuse the programmer art.

Now, I'm basically using the EventSystem and the GraphicRaycaster to fake a mouse position as that UI is rendered on a RenderTexture and is a WorldSpace GUI. Here's a stripped down version of that "mouse forwarding" for your reference. It happens in Update()

 raycastResults.Clear();
 var evsys = UnityEngine.EventSystems.EventSystem.current;
 var ev = new UnityEngine.EventSystems.PointerEventData(evsys);
 ev.position = CalculatePointerEventPosition(Input.mousePosition);

 evsys.RaycastAll(ev, raycastResults);
 for (int i = 0; i < raycastResults.Count; ++i) {
     var cp = raycastResults[i].gameObject.GetComponent<Selectable>();
     if (cp != null) {
         if (lastButton == cp) {
             continue;
             // still active, don't fire again
         }
         DeselectLastSelecatble(ev);
         cp.Select();
         cp.OnSelect(new UnityEngine.EventSystems.BaseEventData(UnityEngine.EventSystems.EventSystem.current));
         lastButton = cp;
     }
 }

This would therefore work for all selectables to at least put them in the "Selected" state. However the raycast made via the EventSystem (which uses the GraphicRaycaster as far as I am aware) seems to outright ignore the Toggle element(s). The toggle elements have the default structure that Unity gives you when creating a new toggle via menu. I can confirm that the calculated fake mouse position is correct though. It works for buttons, for instance.

However, when navigating via arrow keys (axis input) then I can select and toggle these things. Also it works without problems when not rendering to a RenderTexture but directly to screen.

So I wonder, what do I have to do to get that Toggle recognised by the raycaster?