Android load/unload times

I am experiencing a weird problem while porting an ios game to android. The game consists of a MainMenu and several Levels which are all in separate scenes. The levels contain lots of assets, the MainMenu only a few. Now loading times of the levels seems to be as expected (~5-10s) after I converted the textures to RGBA 16 bit. If the game returns to the MainMenu however the load times exceed 10 minutes. And after that it will probably crash (guessing memory). Also we load an empty scene in between levels to let unity do some cleaning. Even that scene takes about 5 minutes to load after playing a level. So I suspect it's more the unloading time than the loading time. Anyone experienced this and knows a way around it?