Unity seems to be taking a long time destroying a scene?

I'm kind of new to Unity, but I've searched and haven't found any answer to this question. I have a game that I've just begun work on that has only a few scenes, none of which has too much going on. But when I call Application.LoadLevel it takes far too long to switch the level. I've come to the conclusion (although its not necessarily true) that the issue lies with destroying the old scene and not loading the new one. Now as I said I'm new to this, but the reasons I suspect this is are:

• can load any of the levels by running them from the editor with minimal load time. It doesn't take a noticeable amount of time to open different scenes either. -If I put print statements in the portions of my code that handle loading things, none print until after the game already seems to have resumed(although Unity could just load assets before this code is run for all I know, so this is probably irrelevant).

Anyway, is it true that the majority of the load time is spent destroying objects of the old scene? Or am I wrong? And either way, is there anything I could do to speed things up a bit? Where is the majority of the load time most likely to be spent?

Thanks!