- Home /
uNET - Send big amount of data over network (How to split send byte[]?)
hi all.
I'm trying to send an object from server to client. the object holds (beside some strings) a byte array which represents an image file. Thus, the objects can be quite huge (depending on the image, e.g. like 300kb). I tried to send it via RPC as custom struct which failed because of the size. Now i'm sending it as NetworkMessage via NetworkServer.SendByChannelToAll on the reliableFragmented channel. I've set the connectionConfig.FragmentSize to 1024. This works for small images. Big images are not being transferred completely, so that only some pixels of the image are shown on the client.
Is there a better way to send this huge amount of data? I thought about serializing the whole object as byte array, split it in chunks and set them one after another, but I'm not sure in how to do that. Is there any example out there?
Cheers, Phineliner
Hello, did u figure that out? i tried the same some while ago but didn't get it to work... could you provide some example code? many thx! $$anonymous$$.
Hello, many thx! This helps me al lot... What is your LobbyController ? Is this your Network$$anonymous$$anager? How to serialized and deserialize is not a problem... but can't find public event UnityAction OnDataComepletelySent; many thx! Wish u a great 2016! $$anonymous$$.
Yes, LoobyController is my Network$$anonymous$$anager. I just added a constant there, saying which is my reliable sequenced channel (set in the inspector). Updated my answer to clarify OnDataCompletelySent.
Answer by phineliner · Dec 16, 2015 at 10:26 PM
I figured it out by myself. Roughly it can be achieved as follows:
I serialize the object into one big byte array which then is sent in several chunks (of 1024 byte) to all clients.
I use ClientRPC to send the chunks.
I use a ReliableSequenced channel / qostype, to ensure the chunks arrive in the right order.
On client side I collect the chunks, add them up to a big byte array again and then I deserialize it back to the object.
Edit:
As I was asked to provide some sample code, here you go:
This is my NetworkTransmission Component:
public class NetworkTransmitter : NetworkBehaviour {
private static readonly string LOG_PREFIX = "[" + typeof(NetworkTransmitter).Name + "]: ";
public const int RELIABLE_SEQUENCED_CHANNEL = MyNetworkManager.CHANNEL_RELIABLE_SEQUENCED;
private static int defaultBufferSize = 1024; //max ethernet MTU is ~1400
private class TransmissionData{
public int curDataIndex; //current position in the array of data already received.
public byte[] data;
public TransmissionData(byte[] _data){
curDataIndex = 0;
data = _data;
}
}
//list of transmissions currently going on. a transmission id is used to uniquely identify to which transmission a received byte[] belongs to.
List<int> serverTransmissionIds = new List<int>();
//maps the transmission id to the data being received.
Dictionary<int, TransmissionData> clientTransmissionData = new Dictionary<int,TransmissionData>();
//callbacks which are invoked on the respective events. int = transmissionId. byte[] = data sent or received.
public event UnityAction<int, byte[]> OnDataComepletelySent;
public event UnityAction<int, byte[]> OnDataFragmentSent;
public event UnityAction<int, byte[]> OnDataFragmentReceived;
public event UnityAction<int, byte[]> OnDataCompletelyReceived;
[Server]
public void SendBytesToClients(int transmissionId, byte[] data)
{
Debug.Assert(!serverTransmissionIds.Contains(transmissionId));
StartCoroutine(SendBytesToClientsRoutine(transmissionId, data));
}
[Server]
public IEnumerator SendBytesToClientsRoutine(int transmissionId, byte[] data)
{
Debug.Assert(!serverTransmissionIds.Contains(transmissionId));
Debug.Log(LOG_PREFIX + "SendBytesToClients processId=" + transmissionId + " | datasize=" + data.Length);
//tell client that he is going to receive some data and tell him how much it will be.
RpcPrepareToReceiveBytes(transmissionId, data.Length);
yield return null;
//begin transmission of data. send chunks of 'bufferSize' until completely transmitted.
serverTransmissionIds.Add(transmissionId);
TransmissionData dataToTransmit = new TransmissionData(data);
int bufferSize = defaultBufferSize;
while (dataToTransmit.curDataIndex < dataToTransmit.data.Length-1)
{
//determine the remaining amount of bytes, still need to be sent.
int remaining = dataToTransmit.data.Length - dataToTransmit.curDataIndex;
if (remaining < bufferSize)
bufferSize = remaining;
//prepare the chunk of data which will be sent in this iteration
byte[] buffer = new byte[bufferSize];
System.Array.Copy(dataToTransmit.data, dataToTransmit.curDataIndex, buffer, 0, bufferSize);
//send the chunk
RpcReceiveBytes(transmissionId, buffer);
dataToTransmit.curDataIndex += bufferSize;
yield return null;
if (null != OnDataFragmentSent)
OnDataFragmentSent.Invoke(transmissionId, buffer);
}
//transmission complete.
serverTransmissionIds.Remove(transmissionId);
if (null != OnDataComepletelySent)
OnDataComepletelySent.Invoke(transmissionId, dataToTransmit.data);
}
[ClientRpc]
private void RpcPrepareToReceiveBytes(int transmissionId, int expectedSize)
{
if (clientTransmissionData.ContainsKey(transmissionId))
return;
//prepare data array which will be filled chunk by chunk by the received data
TransmissionData receivingData = new TransmissionData(new byte[expectedSize]);
clientTransmissionData.Add(transmissionId, receivingData);
}
//use reliable sequenced channel to ensure bytes are sent in correct order
[ClientRpc(channel = RELIABLE_SEQUENCED_CHANNEL)]
private void RpcReceiveBytes(int transmissionId, byte[] recBuffer)
{
//already completely received or not prepared?
if (!clientTransmissionData.ContainsKey(transmissionId))
return;
//copy received data into prepared array and remember current dataposition
TransmissionData dataToReceive = clientTransmissionData[transmissionId];
System.Array.Copy(recBuffer, 0, dataToReceive.data, dataToReceive.curDataIndex, recBuffer.Length);
dataToReceive.curDataIndex += recBuffer.Length;
if (null != OnDataFragmentReceived)
OnDataFragmentReceived(transmissionId, recBuffer);
if (dataToReceive.curDataIndex < dataToReceive.data.Length - 1)
//current data not completely received
return;
//current data completely received
Debug.Log(LOG_PREFIX + "Completely Received Data at transmissionId=" + transmissionId);
clientTransmissionData.Remove(transmissionId);
if (null != OnDataCompletelyReceived)
OnDataCompletelyReceived.Invoke(transmissionId, dataToReceive.data);
}
}
And this is how one would use the component (make sure it is attached to the same game-object of the script that wants to send the data / has a network identity):
//inside a class, derived from network behaviour, which has the NetworkTransmitter component attached...
...
//on client and server
NetworkTransmitter networkTransmitter = GetComponent<NetworkTransmitter>();
...
//on client: listen for and handle received data
networkTransmitter.OnDataCompletelyReceived += MyCompletelyReceivedHandler;
networkTransmitter.OnDataFragmentReceived += MyFragmentReceivedHandler;
...
//on server: transmit data. myDataToSend is an object serialized to byte array.
StartCoroutine(networkTransmitter.SendBytesToClientsRoutine(0, myDataToSend))
...
//on client this will be called once the complete data array has been received
[Client]
private void MyCompletelyReceivedHandler(int transmissionId, byte[] data){
//deserialize data to object and do further things with it...
}
//on clients this will be called every time a chunk (fragment of complete data) has been received
[Client]
private void MyFragmentReceivedHandler(int transmissionId, byte[] data){
//update a progress bar or do something else with the information
}
On how to Serialize an object to byte array, you will find help in other threads.
Hope it helps, Cheers phineliner
Hello , i tried this post to convert object to byte array , but Vector3 and vector2 part throws a error , since it need to be serialized. Can you help me out with that ?
maybe this helps? if not, you maybe should open a new question, since your problem might be specific and depending on your code.
in any case, you could also use your own struct / class to represent Vector2 or Vector3 and make them serializable.
I found a Way :) like you said i created struct to serialize Vec3 and Vec2 Thx mate
Awesome bit of code. It works like a charm! I'm using it to send a serialized mesh over the network. The server creates it programmatically and the clients receive it without issue.
Thanks for the positive feedback! Glad you liked it :)
Wow, really nice. I don't however understand how the transmissionID works. From the looks of it, it is the object that calls SendBytesToClients that is responsible to generate a unique ID. How cant that object be sure that the ID is unique. $$anonymous$$y first idea would be to have a "GetUniqueID" method on the Transmitter or something like that. Would that not make more sense?
Hi jeango,
in the case where I used this component it made more sense to have the calling object generating the unique IDs. Imagine the scenario of having a list of objects that you want to transmit and you want to reassemble this list in the correct order on client site. You do so by iterating the list using a for-loop and calling SendBytesToClient in each iteration. There you use the index of the list-item as unique transmission ID and that way you are able to identify the object that has been received by the client simply by index (since the handler has the id/index as parameter). Of course, the calling object needs to ensure that it does not mix up the IDs in that case. The transmitter class however prevents mixing up by checking whether the respective ID is in use or not.
Quick answer: Check out the following code:
Extension Class to provide serialisation for any object: https://bitbucket.org/stupro_hskl_betreuung_kessler/learnit_merged_ss16/raw/e5244ebb38c8fe70759e632ea4224e48f5ca5833/Unity/LearnIT_$$anonymous$$erged/Assets/Scripts/Util/ObjectSerializationExtension.cs
Example Class which I am serializing: https://bitbucket.org/stupro_hskl_betreuung_kessler/learnit_merged_ss16/raw/e5244ebb38c8fe70759e632ea4224e48f5ca5833/Unity/LearnIT_$$anonymous$$erged/Assets/Scripts/$$anonymous$$odel/Quiz/QuizQuestion.cs
Example usage. Check out the method "DistributeQuizQuestions()": https://bitbucket.org/stupro_hskl_betreuung_kessler/learnit_merged_ss16/raw/e5244ebb38c8fe70759e632ea4224e48f5ca5833/Unity/LearnIT_$$anonymous$$erged/Assets/Scripts/Controller/Quiz/$$anonymous$$PQuizController.cs
Let me know if this helps ;)
I don't understand where it is sending the chunks? Where are the connection/clients?
the code is well organized, well documented and also works super fine tnx for sharing it man, saved me a lot of time, exactly what i needed
Thanks for the positive feedback! Glad you liked it :)
Answer by Ashkan_gc · Sep 29, 2016 at 06:03 PM
You can also use a fragmented (ReliableFragmented) channel to send bigger chunks up to a point.
Answer by kenmarold · Nov 19, 2017 at 06:13 AM
Hello @phineliner. Is it possible to reverse the functionality of this script? Rather than sending a large chunk of data to clients, could I send a large chunk of data to the server? I've been trying to do this with this script but I keep getting the error: Trying to send command for object without authority on the CmdPrepareToReceiveBytes() and CmdReceiveBytes() functions. Any ideas how I might get this to work?
Hi, I have not worked with unity for a while so I'm not very up to date anymore. However my guess would be that you need to change the annotations and names of the functions. Cmd (Command) should be Rpc (ClientRPC) and vice versa. Read about commands and clientrpc in the docs.
you are trying to send data on an networkIdentity without having the authority over the object. a client needs to have authority over the object so it can call a cmd on the server. the player usually has authority over the player, for all other objects the server needs to give authority to the player. it will likely work if you send the data on the player object.
Answer by diffmede · Jan 20, 2019 at 08:41 PM
Hi and thx for the nice example Is there a simple way to prevent sending the data to the local client created when starting a host ?,Hy... Thx for the very nice example. Is there a possibility to prevent sending the data to the local client created starting the host ?
I think I don't understand your question. You usually do trigger the data transfer explicitly by invoking networkTransmitter.SendBytesToClientsRoutine
In my network architecture there is local client connected to the server because the server is also a client (not dedicated). But the local client does not need the scene data because he is hosting it. Using networkTransmitter.SendBytesToClientsRoutine will send the data to all clients including this local client.
But I could solve this problem by using [TargetRCP] and sending the data specifically to the clients.
Ah, now I see. Well, glad you figured it out yourself :)