Unity not releasing ports correctly?

I am building a server with a client application. Everything seems to work, but seemingly at random, unity decides to NOT close a port on stopping the project in the editor or when running (build for windows).

Most of the times i have this problem once or 2ce (on first boot. If it correctly works the first time starting and stopping the editor project, it will work for the rest of the runs.)

However, this is starting to get on my nerves as switching ports on both applciations is a pain in the ass, and the final build should just work without this problem.

Here is my server application (Note that no client has to connect for this to happen):

     public class ServerScript : MonoBehaviour {    
 // Use this for initialization
 void Start () {
     instance = this;
     NetworkConnectionError error = Network.InitializeServer(500,26556,false);
     if(error != NetworkConnectionError.NoError){
         Debug.Log("Server couldn't run: " + error);
     }
     Debug.Log("[Server]Server started: "+GetIP());
 }
 #region connect/disconnect
 void OnPlayerConnected(NetworkPlayer user){
     Debug.Log("[Server]Connected: "+ user.externalIP);
 }
 
 void OnPlayerDisconnected(NetworkPlayer user){
     //remove this players RPC calls, he might have none but just make sure.
     Debug.Log("[Server]Disconnected: "+ user.externalIP);
     Network.RemoveRPCs(user);
     Network.DestroyPlayerObjects(user);
 }
 
     void OnDestroy(){
         Network.Disconnect();
     }
     #endregion
     string GetIP(){
         string strMachineName = Dns.GetHostName();
         IPHostEntry ipHost = Dns.GetHostByName(strMachineName);
         IPAddress IP = null;
         foreach(IPAddress ip in ipHost.AddressList){
             if(ip.AddressFamily == System.Net.Sockets.AddressFamily.InterNetwork){
                 IP = ip;    
             }
         }    
         return IP.ToString();
     }
     
     public void returnDataToUser(string JsonObject, NetworkPlayer user, LoLCall.CallType calltype){
         //make RPC call back to user to deliver data
         Debug.Log("[Server]Returning to "+user.externalIP+": " + JsonObject);
         foreach (NetworkPlayer player in Network.connections) { //make sure this user didn't bugger out in the mean time!
             if(player.Equals(user))networkView.RPC("DataResponse", user, Utils.GetBytes(JsonObject), calltype);
         }
     }
     
 
     #region RPC
     //the user calls this when he wants data, the cache will return data with an RPC back.
     [RPC]
     void DataRequest(string JsonCall, NetworkMessageInfo info){
         Debug.Log("[Server]User made a data request: " + JsonCall);
         //cache.makeRequest(JsonCall, info.sender);
     }
     
     [RPC]
     void DataResponse(byte[] JsonResult){//, string callType){
         //THIS IS A STUB TO MAKE RPC WORK!
     }
     #endregion
 }

And the errors are:

 Failed to initialize network interface. Is the listen port already in use?
 Server couldn't run: CreateSocketOrThreadFailure

Does anyone have any idea as to why this problem keeps popping up? In general there is not a lot of documentation regarding unity's Network.* part, i wish there was more troubleshooting on this subject.

Thanks in advance, Smiley