Scene Load Time with many game objects

Hi, My apologies if this has been asked, I could not find anything exactly fitting what I'm asking. I have a project that I'm working on where there is a map, different regions of the map are divided using different size buttons to fill the irregular borders.
So in the unity editor I have nested game objects according to area of the map and then the specific region. Under the region I have buttons that have been added to the scene to map the border. There are, at this time over 400 regions with about an average of 30 buttons each; so the scene contains 12,000 buttons at this point.
When I 'play' the scene in Unity it takes about 5 minutes to load, this is without anything else being done (commented out any code that would have been running). If I generate an executable for this scene and run it (it is a windows desktop game), the executable takes about 2 minutes to load - again with nothing being done in the code, just the objects being loaded. My question is, I understand that I have a lot of objects in this scene (and more to add, my guess is it will reach around 20,000 by the time I'm done), is that load time to be expected? It seems very long for simply building the graphical part of the scene. Is there any way I can decrease this time short of using significantly less buttons/objects in the scene? I certainly don't have an expectation it will load in a second or two but my goal/hope is to get it to load within a minute or two with all the mapping done and all the behind the scenes code running. Any help would be greatly appreciated on this. Thanks erik