NetworkManager.Shutdown does not behave as advertised - HOW do you completely reset all UNET functionality?

My issue is really very simple, and I've seen other people with similar problems, however no real solution that works for me.

I have a game with three scenes: MainMenu, Lobby, and Game.

Lobby-scene is used as a simplified "find quick game"-lobby with a NetworkManager deriving from NetworkLobbyManager. Game-scene works both in multiplayer mode and in singleplayer mode. MainMenu-scene should be self explanatory.

I'm having issues when leaving to MainMenu after initializing the NetworkManager system. If I just leave without trying to destroy network functionality I have issues when returning to Lobby and there seems to be a lot to keep track of and this solution feels generally unstable.

So I think "fine, I'll just completely reset all network functionality." The flow works perfect when I enter Lobby from MainMenu the first time, so if I can just return to that state (a "clean" MainMenu), then my problems are gone.

This is pretty much how NetworkManager.Shutdown is explained in the docs):

"Shuts down the NetworkManager completely and destroy the singleton. This is required if a new NetworkManager instance needs to be created after the original one was destroyed. The example below has a reference to the GameObject with the NetworkManager on it and destroys the instance before calling Shutdown() and switching scenes."

 Destroy(NetworkManagerGameObject);
 NetworkManager.Shutdown();
 Application.LoadLevel("empty");

The problem with this is that it doesn't work. This is my code:

     public void ReturnToMainMenu()
     {
         if (IsFading) return;
         IsFading = true;
         _fader.FadeOut(0.7f, () =>
         {
             Destroy(NetworkManager.singleton.gameObject);
             NetworkManager.Shutdown();
             SceneManager.LoadScene("MainMenu");
             _fader.FadeIn(0.7f, () =>
             {
                 IsFading = false;
             });
         });
     }

What happens is that a new NetworkManager is created when I enter MainMenu and starts doing all kinds off stuff, which really messes up the state of things.
I just really, really, really would like to enter a clean, Unet-less state after once initiating the NetworkManager machine. Is this possible? If not, why?