Scrollable, clickable overworld map

I'm trying to implement a scrollable overworld map that will also accept mouse clicks. I have a map game object whose size is larger than the viewport of my device. That game object has Unity's ScrollRect component attached to it, to handle the scrollable layer (the scrolling is clamped, but it can be scrolled in any direction).

Underneath this game object is the object to be scrolled (a single Image script), and a few "zones" that should be clickable to travel to those zones in the game.

To give you an idea of the game object hierarchy, it looks like this:

 Main Camera (Physics2DRaycaster)
 ui_root (Canvas, GraphicsRaycaster)
   map (ScrollRect)
     map_image (Image)
       zones (empty)
         forest (PolygonCollider2D, ZoneClickHandler)
           title (Text)
         cave (PolygonCollider2D, ZoneClickHandler)
           title (Text)

The tricky part here comes from the fact that the zones aren't simply rectangular areas, so I can't use Unity's default Button component to check if a zone has been clicked; the zones on the map are actually concave polygons. To get around this, I implemented each zone's root game object as a 2D polygon collider with a custom script to implement IPointerClickEvent when the collider is clicked.

I should also note that each zone is not its own sprite. I've seen similar implementations that check the alpha channel on a sprite using a custom ICanvasRaycastFilter, but that won't work here since the map is a single sprite, and I'm simply defining clickable regions on that sprite.

Unfortunately, this approach doesn't seem to be working correctly (sorry for burying the lede, but I'm sure you already realized this!). In fact, what happens is that my ZoneClickHandler#OnPointerClick method is only fired when the mouse is clicked over the "title" game object's RectTransform, inside each zone (which is much smaller than its containing zone game object and polygon collider, and rectangular). I have no idea why that would be the case, since the ZoneClickHandler component is attached to the zone game object itself, not its child "title" game object.

I wouldn't mind hearing approaches that are both inline and not inline (e.g., throwing out the idea that I can mix-and-match UI- and non-UI-related objects) with the way I'm trying to get this going.