New UI system too slow for a minimap

I've built a mini-map using the new UI system. I instantiated a panel for each block of my map. I'm currently have a 200 by 200 map, so that's about 40 000 panels.

Unity's UI system can't cope with that at all. Each time I modify something on the UI, for instance when I draw a string containing the player's coordinate on the screen, it refreshes the UI and make the game lagging a lot. "Unaccounted time between start of frame and Camera.Render" takes about 80% of the processor time in the profiler.

How could I optimise my mini-map ? Should I reimplement a panel to add a custom openGL drawing on top of it ? Is that even possible ?

I also need to update the player's position on the minimap, but that's not an option since I can't even update the UI without the game lagging. (We're talking about 500ms each time the player moves because it updates it's coordinates on the screen).

I tried to move directly a panel showing the player position, so I don't have to update all the minimap panels each time the player moves. However, Unity keeps updating the all UI, and thus, creating a lot of lag, each time I move a panel.

(By the way, using a camera placed on top of the scene and draw it on the UI to make a minimap is not an option for me because my map is too big and the minimap has to show it all at once)