uLink Server p2p issue

Summary

Server p2p hand over does not seem to work correctly. The first player that connects is the only player that ever appears to actually get handed over. Read on for more detail. Note: when I say "master server" in this post, I am NOT referring to uLink's master server.

Background

I'm working on a game using uLink that has a somewhat complex server architecture. The servers are authoritative. Basically, each player has their own auth server that runs the game logic specifically for them. They are delegated a server when first loading into the game by the Master server.

So, when a player connects to the game, a Master Server finds a server from a pool, and hands over the player to that server via p2p.

Here is the code that does this. The server has a stack of servers that it pops from and then uses the p2p connection to that server to hand over. _p2p is the uLink.NetworkP2P object that allows us to connect to each server in the pool.

Master server code:

 void uLink_OnPlayerConnected(NetworkPlayer player)
 {
   if (_availableServers.Count == 0)
   {
     throw new UnityException("No servers left in pool for player to connect to.");
   }
   var networkPeer = _availableServers.Pop();  // Get fresh server from the pool to connect to
   Debug.Log("Player connected; assigning server for spawning. Player: " + player + ", Server: " + networkPeer);
   _playerServerMap.Add(player, networkPeer);  // Add the player/server to a map to track which servers belong to which players
   _p2p.Handover(player, networkPeer);  // Hand over the player to the fresh server we just popped off the stack
 }

Ok so now that the player has been handed over, the server from the pool instantiates the owner/creator/proxy prefabs just like the uLink simple server script. This all happens in the onPlayerConnected event. Here is that code:

Player server code:

 void uLink_OnPlayerConnected(uLink.NetworkPlayer player)
 {
   _player = player;
   Debug.Log("Player assigned to this server: " + _player);
   Quaternion rotation = Quaternion.Euler(new Vector3(0, 0, 0));
   uLink.Network.Instantiate(player, ProxyPrefab, OwnerPrefab, ServerPrefab, StartPosition, rotation, 0, playerName);
 }

At first this all appears to work fine. I have successfully loaded up two players and two servers, and the master server appears to do the hand overs correctly.

The Problem

If you look at the logging statements, the Master server will correctly print out the following after two players connect:

 "Player connected; assigning server for spawning. Player: Player Client(1), Server: 127.0.0.1:21001"
 
 "Player connected; assigning server for spawning. Player: Player Client(2), Server: 127.0.0.1:21000"

The ports just being the ports for each server from the pool. So player one gets connected to the server at port 21001 and player 2 gets connected to the server at port 21000. From debugging the data structures look perfect on the master server side.

However, if you look at the logging statements for the player servers when the handover is performed, in the code above, here is what they both say:

  Player assigned to this server: Player Client (1)

Which means that the uLink_OnPlayerConnected event is sending Player (1) both times, even though the handover is specifically sending different players to different players (see the master server code above again).

The problem manifests when I try to do cleanup in particular. After a player disconnects the player server tells the master server that the player is gone, and the master server finds that player in the dictionary and returns their server to the available servers stack so someone else can use it. However, since both player servers think they are using the same player, the cleanup operation fails for the second player. I imagine I'd run into even more issues like this as I continue to develop my game.

EDIT: I should also note that I'm testing this all on my local machine. So I have a master server running, two player servers for the pool, and two clients. Maybe it has something to do with being on the same machine?

Either this is a bug or I have some fundamental concept wrong about uLink's p2p logic.

Thanks for any troubleshooting tips or help!

EDIT2: Here's some interesting additional info. Instead of using the uLink_OnPlayerConnected in the player server, I instead made an RPC method, SpawnPlayer, that did the logic of the previous onPlayerConnected, which instantiates the player proxy/creator/owner prefabs. So after the handover is performed on the master server, it calls this RPC on the player server to instantiate the prefabs, and it's one argument is the player object. Recall that the player object on the master server is correct up to the handover point. So basically I'm forcing the player server to instantiate using Player (2).

The result was that it loaded a proxy into the scene only, so I can't control it; it's as if it's another player. Which makes some sense since it appears to think player (1) is the true owner of the client.

Another interesting bit. I turned on some additional uLink logging and found that it really does seem like it's sending Player (2) in the handover.

When the second player connects this is the master servers output:

 P2P handover response #1 redirecting Player Client (2)

But then the player server gets Player Client (1)! What the heck!

Anyway, hope the additional info is helpful. Thanks again.

EDIT3: Sorry for all the edits but I'm trying to troubleshoot on my own while I'm waiting for moderation. I also noticed this in the logs for the second server when it receives the handover request:

 Server got client connect request as part of a handover, Player Client(2)
 Server allocated new Player Client (1)

So it does get the handover for player 2, and then promptly just creates a new player client. That seems...wrong. The investigation continues...