TCP Socket Async BeginSend never happens

After doing some more testing, I've come to the conclusion it's a bug/effect on threads while running in Unity Editor. Here's how I came to that conclusion:

I noticed the problem occurred more often the first time opening the Unity project and starting the game. Stopping, and then starting, usually would work. It'd always work within 2-3 tries and continue to work for at least another dozen or so attempts. That could of course indicate some kind of race condition or threading concurrency issue in my own code, so I did the following: 1) Made a very stripped down tcplistener/tcpclient project in Unity, eliminating any caching/recycling of byte[] arrays or other things that may inadvertently affect the async or overall performance. 2) I tested this new project both in editor, and as a standalone build to check for outcome.

This of course requires unity, though it could probably just as easily be adopted to a .net console application for further evaluation. The project consisted of one scene, with a game camera the following three scripts attached to the camera. You need to drag/drop the TCPClient and TCPServer references onto ConnectGUI once attached to the gameobject. The code:

ConnectGUI.cs

 using UnityEngine;
 using System.Collections;

 public class ConnectGUI : MonoBehaviour {

 public enum ConnectionState
 {
     NotConnected,
     AttemptingConnect,
     Connected
 }

 public TCPClient client;
 public TCPServer server;

 // Use this for initialization
 void Start () 
 {
     client.connectState = ConnectionState.NotConnected;
 }
 
 // Update is called once per frame
 void Update () {
 
 }

 void OnGUI()
 {
     GUI.Label(new Rect(10, 10, Screen.width - 20, 20), client.connectState.ToString());

     if (client.connectState == ConnectionState.NotConnected)
     {
         if (GUI.Button(new Rect(Screen.width * 0.5f - 200, Screen.height * 0.5f - 40, 400, 80), "Connect"))
         {
             server.StartServer();
             System.Threading.Thread.Sleep(10);
             client.StartConnect(); 
         }
     }
 }
 }

TCPClient.cs

 using UnityEngine;
 using System.Collections;
 using System.Net;
 using System.Net.Sockets;

 public class TCPClient : MonoBehaviour {

 public ConnectGUI.ConnectionState connectState;
 Socket m_clientSocket;
 byte[] m_readBuffer;

 void Start()
 {
     connectState = ConnectGUI.ConnectionState.NotConnected;
     m_readBuffer = new byte[1024];
 }

 public void StartConnect()
 {
     m_clientSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
     try
     {
        System.IAsyncResult result = m_clientSocket.BeginConnect("127.0.0.1", 10000, EndConnect, null);

        bool connectSuccess = result.AsyncWaitHandle.WaitOne(System.TimeSpan.FromSeconds(10));

        if (!connectSuccess)
        {
            m_clientSocket.Close();
            Debug.LogError(string.Format("Client unable to connect. Failed"));
        }
     }
     catch (System.Exception ex)
     {
         Debug.LogError(string.Format("Client exception on beginconnect: {0}", ex.Message));
     }

     connectState = ConnectGUI.ConnectionState.AttemptingConnect;
 }

 void EndConnect(System.IAsyncResult iar)
 {
     m_clientSocket.EndConnect(iar);

     m_clientSocket.NoDelay = true;

     connectState = ConnectGUI.ConnectionState.Connected;

     BeginReceiveData();

     Debug.Log("Client connected");
 }

 void OnDestroy()
 {
     if (m_clientSocket != null)
     {
         m_clientSocket.Close();
         m_clientSocket = null;
     }
 }

 void BeginReceiveData()
 {
     m_clientSocket.BeginReceive(m_readBuffer, 0, m_readBuffer.Length, SocketFlags.None, EndReceiveData, null);
 }

 void EndReceiveData(System.IAsyncResult iar)
 {
     int numBytesReceived = m_clientSocket.EndReceive(iar);

     ProcessData(numBytesReceived);

     BeginReceiveData();
 }

 void ProcessData(int numBytesRecv)
 {
     string temp = TCPServer.CompileBytesIntoString(m_readBuffer, numBytesRecv);
     
     Debug.Log(string.Format("Client recv: '{0}'", temp));

     byte[] replyMsg = new byte[m_readBuffer.Length];
     System.Buffer.BlockCopy(m_readBuffer, 0, replyMsg, 0, numBytesRecv);
     
     //Increment first byte and send it back
     replyMsg[0] = (byte)((int)replyMsg[0] + 1);
     
     SendReply(replyMsg, numBytesRecv);
 }

 void SendReply(byte[] msgArray, int len)
 {
     string temp = TCPServer.CompileBytesIntoString(msgArray, len);

     Debug.Log(string.Format("Client sending: len: {1} '{0}'", temp, len));

     m_clientSocket.BeginSend(msgArray, 0, len, SocketFlags.None, EndSend, msgArray);
 }

 void EndSend(System.IAsyncResult iar)
 {
     m_clientSocket.EndSend(iar);

     byte[] msg = (iar.AsyncState as byte[]);

     string temp = TCPServer.CompileBytesIntoString(msg, msg.Length);

     Debug.Log(string.Format("Client sent: '{0}'", temp));

     System.Array.Clear(msg, 0, msg.Length);
     msg = null;
 }
 }

TCPServer.cs

 using UnityEngine;
 using System.Collections;
 using System.Net;
 using System.Net.Sockets;

 public class TCPServer : MonoBehaviour 
 {
 public enum TestMessageOrder
 {
     NotConnected,
     Connected,
     SendFirstMessage,
     ReceiveFirstMessageReply,
     SendSecondMessage,
     ReceiveSecondMessageReply,
     SendThirdMessage,
     ReceiveThirdMessageReply,
     Error,
     Done
 }

 protected TcpListener m_tcpListener;
 protected Socket m_testClientSocket;
 protected byte[] m_readBuffer;
 [SerializeField]
 protected TestMessageOrder m_testClientState;

 public void StartServer()
 {
     m_tcpListener = new TcpListener(IPAddress.Any, 10000);
     m_tcpListener.Start();

     StartListeningForConnections();
 }

 void StartListeningForConnections()
 {
     m_tcpListener.BeginAcceptSocket(AcceptNewSocket, m_tcpListener);
     Debug.Log("SERVER ACCEPTING NEW CLIENTS");
 }

 void AcceptNewSocket(System.IAsyncResult iar)
 {
     m_testClientSocket = null;
     m_testClientState = TestMessageOrder.NotConnected;
     m_readBuffer = new byte[1024];

     try
     {
         m_testClientSocket = m_tcpListener.EndAcceptSocket(iar);
     }
     catch (System.Exception ex)
     {
         //Debug.LogError(string.Format("Exception on new socket: {0}", ex.Message));
     }

     m_testClientSocket.NoDelay = true;
     m_testClientState = TestMessageOrder.Connected;

     BeginReceiveData();
     SendTestData();

     StartListeningForConnections();
 }

 void SendTestData()
 {
     Debug.Log(string.Format("Server: Client state: {0}", m_testClientState));

     switch (m_testClientState)
     {
         case TestMessageOrder.Connected:
             SendMessageOne();
             break;

         //case TestMessageOrder.SendFirstMessage:
             //break;

         case TestMessageOrder.ReceiveFirstMessageReply:
             SendMessageTwo();
             break;

         //case TestMessageOrder.SendSecondMessage:
             //break;

         case TestMessageOrder.ReceiveSecondMessageReply:
             SendMessageTwo();
             break;

         case TestMessageOrder.SendThirdMessage:
             break;

         case TestMessageOrder.ReceiveThirdMessageReply:
             m_testClientState = TestMessageOrder.Done;
             Debug.Log("ALL DONE");
             break;

         case TestMessageOrder.Done:
             break;

         default:
             Debug.LogError("Server shouldn't be here");
             break;
     }
 }

 void SendMessageOne()
 {
     m_testClientState = TestMessageOrder.SendFirstMessage;
     byte[] newMsg = new byte[] { 1, 100, 101, 102, 103, 104 };

     SendMessage(newMsg);
 }

 void SendMessageTwo()
 {
     m_testClientState = TestMessageOrder.SendSecondMessage;
     byte[] newMsg = new byte[] { 3, 100, 101, 102, 103, 104, 105, 106 };

     SendMessage(newMsg);
 }

 void SendMessageThree()
 {
     m_testClientState = TestMessageOrder.SendThirdMessage;
     byte[] newMsg = new byte[] { 5, 100, 101, 102, 103, 104, 105, 106, 107, 108 };

     SendMessage(newMsg);
 }

 void SendMessage(byte[] msg)
 {
     string temp = TCPServer.CompileBytesIntoString(msg);

     Debug.Log(string.Format("Server sending: '{0}'", temp));

     m_testClientSocket.BeginSend(msg, 0, msg.Length, SocketFlags.None, EndSend, msg);
 }

 void EndSend(System.IAsyncResult iar)
 {
     m_testClientSocket.EndSend(iar);

     byte[] msgSent = (iar.AsyncState as byte[]);
     string temp = CompileBytesIntoString(msgSent);

     Debug.Log(string.Format("Server sent: '{0}'", temp));
 }

 void BeginReceiveData()
 {
     m_testClientSocket.BeginReceive(m_readBuffer, 0, m_readBuffer.Length, SocketFlags.None, EndReceiveData, null);
 }

 void EndReceiveData(System.IAsyncResult iar)
 {
     int numBytesReceived = m_testClientSocket.EndReceive(iar);

     ProcessData(numBytesReceived);

     BeginReceiveData();
 }

 void ProcessData(int numBytesRecv)
 {
     string temp = TCPServer.CompileBytesIntoString(m_readBuffer, numBytesRecv);

     Debug.Log(string.Format("Server recv: '{0}'", temp));

     byte firstByte = m_readBuffer[0];

     switch (firstByte)
     {
         case 1:
             Debug.LogError(string.Format("Server should not receive first byte of 1"));
             m_testClientState = TestMessageOrder.Error;
             break;

         case 2:
             m_testClientState = TestMessageOrder.ReceiveSecondMessageReply;
             break;

         case 3:
             Debug.LogError(string.Format("Server should not receive first byte of 3"));
             m_testClientState = TestMessageOrder.Error;
             break;

         case 4:
             m_testClientState = TestMessageOrder.ReceiveThirdMessageReply;
             break;

         case 5:
             Debug.LogError(string.Format("Server should not receive first byte of 5"));
             m_testClientState = TestMessageOrder.Error;
             break;

         default:
             Debug.LogError(string.Format("Server should not receive first byte of {0}", firstByte));
             m_testClientState = TestMessageOrder.Error;
             break;
     }

     SendTestData();
 }

 void OnDestroy()
 {

     if (m_testClientSocket != null)
     {
         m_testClientSocket.Close();
         m_testClientSocket = null;
     }

     if (m_tcpListener != null)
     {
         m_tcpListener.Stop();
         m_tcpListener = null;
     }
 }

 public static string CompileBytesIntoString(byte[] msg, int len = -1)
 {
     string temp = "";

     int count = len;

     if (count < 1)
     {
         count = msg.Length;
     }

     for (int i = 0; i < count; i++)
     {
         temp += string.Format("{0} ", msg[i]);
     }

     return temp;
 }
 }

What this does is starts a TcpListener, and begins an async connection socket accept. Then a client socket is created and connects as a tcp socket (on port 10000, of 127.0.0.1). It turns off Nagle's algorithm and the server sends a first message. The client receives the message, increments the first byte from 1->2 and returns the original message. Server then receives that message, and sends another message starting with 3. Client receives, increments 3->4 and echos back the rest of the message. Server then receives that, and sends a 3rd and last message starting with 5. Client turns 5->6 and sends back message. Once that occurs, the server prints "ALL DONE". Both server and client should print to log the various message contents (not always in the same order due to the nature of threading).

If for some reason "ALL DONE" is not printed, then the experiment has failed.

Running this in Unity Editor, it failed 10/10 on the first run, when immediately run after opening the editor. Subsequent attempts to run it resulted in mixed success for the 2nd and 3rd attempts. By the 4th attempt, I have no recorded failures.

I then compiled the project as a standalone program, and repeated the same number of attempts. Since it was reliant on the "ALL DONE" in the log, output.log was checked for "ALL DONE" and was found each time.

So, unless I'm misinterpretating the results, there is a problem either in Unity Editor's or its underlying mono version that is mucking about with threads which causes tcp async read/writes to fail in some capacity. However in the standalone builds, whatever that something is, thankfully, does not seem to appear to be a problem at least as far as testing on Windows allows.

I fully admit the testing was limited with only about 40 runs each but the results were significantly different, though I am too lazy to calculate actual significance. I am puzzled and a still bit concerned that it may be my own flawed implementation, since something like this is not more widespread; however Unity's own networking relies mainly on RPC calls, and most middleware fully embrace an exclusive UDP based network option.

If there is some fundamental flaw present, please let me know, otherwise I hope this may help some lost soul (as I was for almost two weeks) in the future as there is little to no searchable results on this topic. This was all done in Unity 4.6.1f1, but also tested by a friend in the present Unity 5 Beta (unsure of current beta version number).

Personally, while this is extremely annoying, I feel I can ignore this as an editor-only problem with little-to-no potential to impact actual players playing a compiled version. It will be something to heavily test once builds are regularly happening.