How to Destroy NetworkLobbyManager

I've been having some issues with NetworkLobbyManager. To work round, I decided to try to Destroy and recreate the NetworkLobbyManager each time the player comes back to the main screen.

I use the code below:

 private void ResurrectNetworkManager() {
 
     if (lobbyManager != null) {
         Debug.Log("killing"+lobbyManager.gameObject);
         DestroyObject(lobbyManager);
         Debug.Log("creating again");
         lobbyManager = Instantiate(lobbyManagerPrefab);
     }
 }

It seems to Destroy fine. But when the instantiate happens again, I get the following:

Multiple NetworkManagers detected in the scene. Only one NetworkManager can exist at a time. The duplicate NetworkManager will not be used. UnityEngine.Object:Instantiate(GameObject) MainMenuManager:ResurrectNetworkManager() (at Assets/MainMenuManager.cs:35) MainMenuManager:Awake() (at Assets/MainMenuManager.cs:24)

Is there some trick I need to do to fully destroy the NetworkManager?