Resource Unload - Doesn't clear memory fully

Hello,

I'm trying dynamically load and then unload a prefab at runtime via Resource Load/Unload. But every time I unload the prefab, I don't get all my memory back:

As you can see, at the start, the app is running normally. I then load in the prefab which then causes the large spike as expected, I then remove the prefab, but the memory doesn't go back down to the starting height. I only get about 25% of my memory back.

Why is this?

How I load:

 //get the prefab directory and load
 request = Resources.LoadAsync( "_SteamingPrefabs/" + GetPrefabName(story, page) );
 
 //visual UI
 while(request.isDone == false){
 
     var _f : float = request.progress * 100;
     progressWheel.SetPercentage( _f );
     yield;
 }
 
 //resource object itself
 loadedResource = request.asset;
 
 //make clone of prefab
 loadedPrefab = Instantiate(loadedResource) as GameObject;

How I unload:

 request = null;
 
 children = loadedPrefab.GetComponentsInChildren.<Transform>(true);
 for(var i = children.length-1; i >= 0; i--){
 
     UnityEngine.Object.Destroy(children[i].gameObject);
     children[i] = null;
 }
 children = null;
 UnityEngine.Object.Destroy(loadedPrefab);
 
 loadedResource = null;
 loadedPrefab = null;
 
 yield WaitForEndOfFrame();
 yield WaitForEndOfFrame();
 
 Debug.LogWarning("[SERIAL] Unloading Resources");
 yield Resources.UnloadUnusedAssets();
 
 yield WaitForEndOfFrame();
 Resources.UnloadUnusedAssets();

Is there something I'm missing here? I'm searching endless forums and threads, and I still don't know what I'm doing wrong??

Whats interesting, is that if I load the prefab back in - the memory goes back to the original height. Etc.