Physics.Raycast only finds some objects

I have a GameObject that implements IDragHandler and IDropHandler from UnityEngine.EventSystems.

In OnDrop, I want to check, has this object been dropped in front of another object. I'm using Physics.Raycast, but it only returns true in some cases. EDIT: I use screen point to ray as direction for my ray, and this transform as origin for the ray of the Raycast.

My code:

     public void OnDrop(PointerEventData eventData)
     {
             var screenRay = Camera.main.ScreenPointToRay(new Vector3(eventData.position.x, eventData.position.y, 0.0f));
             var thisToObjectBehind = new Ray(transform.position, screenRay.direction);

             Debug.DrawRay(thisToObjectBehind.origin, thisToObjectBehind.direction, Color.yellow, 20.0f, false);

             RaycastHit hit;
             if (Physics.Raycast(thisToObjectBehind , out hit))
             {
                 Debug.LogFormat("Dropped in front of {0}!", hit.transform;
             }
     }

I'm using a perspective camera. When the object is dropped in front of objects straight forward from the screen/camera, Physics.Raycast sometimes returns true, but "hit" contains this, not the object behind this. Sometimes it returns false. None of these two results are expected, there are objects behind this that should be possible to Raycast.

When the object is dropped in front objects in the outskirts of the camera view, Physics.Raycast succeeds in finding the objects behind.

The Debug ray looks fine, it is drawn from the object I dropped, backwards to the object it should hit.