Unable to free resources from memory

I have a game with several levels. Each time I load or switch to a new level, the textures and resources from the previous level remain in memory. To the point that when I'm on the final level, it's as if I have loaded every resource in my game and then I get a crash with "Out of memory" error. I'm not using Resources.Load or AssetBundles, and Resources.UnloadUnusedAssets doesnt work. I just plop my game objects with textures into the scene. Is this a bad thing to do? Do I have to dynamically load my textures and resources to make sure that they are freed when loading a new level? I wouldn't imagine so, I thought this is what Unity was for. But this is causing me huge problems. Any help is appreciated.