Wayback Machinekoobas.hobune.stream
May JUN Jul
Previous capture 12 Next capture
2021 2022 2023
1 capture
12 Jun 22 - 12 Jun 22
sparklines
Close Help
  • Products
  • Solutions
  • Made with Unity
  • Learning
  • Support & Services
  • Community
  • Asset Store
  • Get Unity

UNITY ACCOUNT

You need a Unity Account to shop in the Online and Asset Stores, participate in the Unity Community and manage your license portfolio. Login Create account
  • Blog
  • Forums
  • Answers
  • Evangelists
  • User Groups
  • Beta Program
  • Advisory Panel

Navigation

  • Home
  • Products
  • Solutions
  • Made with Unity
  • Learning
  • Support & Services
  • Community
    • Blog
    • Forums
    • Answers
    • Evangelists
    • User Groups
    • Beta Program
    • Advisory Panel

Unity account

You need a Unity Account to shop in the Online and Asset Stores, participate in the Unity Community and manage your license portfolio. Login Create account

Language

  • Chinese
  • Spanish
  • Japanese
  • Korean
  • Portuguese
  • Ask a question
  • Spaces
    • Default
    • Help Room
    • META
    • Moderators
    • Topics
    • Questions
    • Users
    • Badges
  • Home /
avatar image
2
Question by Brain-_-Fail · May 08, 2017 at 02:47 PM · videotexturesrpcstreaming

Unity simple client server video streaming using RPC calls to send webcamTexture

, I have created a simple application in unity that acts as a client as well as a server.The server runs on android device while the client on a laptop/desktop.What the application does is the client captures a photo from its device webcam then converts it to JPG byte array and sends this to the server using an RPC call, the server then renders the texture, this process happens quickly and repeatedly thus simulating a live video stream.The problem is that i am getting poor framerate maybe around 8frames/s.Can someone help me get it right maybe improve the framerate to around 30 or something above.Here is the algorithm:

The client captures a shot from webcam uses (WebCamTexture.GetPixels() in

unity)

Then converts it to JPG using Texture2D.EncodeToJPG().

Then send this byte array which is on average 4Kb in size and never exceeds 4.5Kb

The server then loads this texture using Texture2D.LoadImage(textureBytes)

Here is the code now(Just included the relevant portion, note not tested might not run !) :

    using System;
    using System.Collections;
    using System.Collections.Generic;
    using UnityEngine;
    using UnityEngine.UI;

    public class Media : MonoBehaviour
    {


 [SerializeField]
 private RawImage videoPanel;                          // The rawimage on which the video stream will be showed


 private WebCamTexture liveRec;
 private bool routineRunning;
 private bool texelsLoaded = true;                    
 private bool noSignal;                               
 private Texture2D image;
 private byte[] texelsBinary;
 private byte[] textureBytes;
 private int qualityControl;                      
 private bool asyncLoadRunning;                            
 private int  rpcBufferCount;


 internal static int width;
 internal static int height;

 public int bufferLimit = 60;                             
 private Toggle serverswitch;
 private Toggle clientswitch;


 // Use this for initialization


 void Start()
 {

     texelsBinary = new byte[2];
     textureBytes = new byte[2];


     qualityControl = 70;


     liveRec = new WebCamTexture();
     liveRec.requestedHeight = 1; 
     liveRec.requestedWidth = 1; 
     liveRec.anisoLevel = 1;
     liveRec.mipMapBias = 1.5f;

     if (!(Application.platform == RuntimePlatform.Android)) { liveRec.Play(); }
     liveRec.anisoLevel = 1;
     width = liveRec.width;
     height = liveRec.height;
     image = new Texture2D(liveRec.width, liveRec.height);




 }




 void Update()
 {
   
             if(serverswitch.isOn && ProceduralLink.isConnected())           // Means if the application is acting a s a server

             {
                 if (!asyncLoadRunning)
                 {
       
                   asyncLoadRunning = true;
                   StartCoroutine(serverAsyncLoader());
                 }
             }                        
    

     if (clientswitch.isOn && !routineRunning)                // Means the application is acting as a client take photos and send them to the server app on android
         
     {
         routineRunning = true;
         StartCoroutine(TakePhoto());
     }


 }


 IEnumerator TakePhoto()
 {

     yield return new WaitForSeconds(0);


    if (rpcBufferCount > bufferLimit) { rpcBufferCount = 0; Network.RemoveRPCs(ProceduralLink.clientViewId); }  //RPC calls seem to buffer up so i clear them up otherwise on the server the video stream is left far behind the live stream


    makeNextShotReady();
    sendTexels();
    rpcBufferCount += 1;             
      
    routineRunning = false;
     
 }


 // server RPC

 [RPC]

 void loadTexels(byte[] buffer)

 {

   if (!noSignal)

     {
         textureBytes = buffer;
     }

 }


 private void sendTexels()

 {
     
         try   { GetComponent<NetworkView>().RPC("loadTexels", RPCMode.Server, texelsBinary); }
         catch { return; }
            
 }


 private void makeNextShotReady()

 {
     image.SetPixels32(liveRec.GetPixels32());
     texelsBinary = image.EncodeToJPG(qualityControl);
 }


 IEnumerator serverAsyncLoader()
 {   
     yield return new WaitForSeconds(0);
     image.LoadImage(textureBytes);
     videoPanel.texture = image;
     asyncLoadRunning = false;
 }





     }



Some facts:

1)> The resolution of the camera device i have lowered down to the lowest possible and the final texture size that i get when it is encoded into JPG is always around 4KB so size is nopt an issue.

2)> The server and client apps are connected through LAN where the bandwidth is dedicated to them forever from my home router.

3)> I have a searched on the internet they seem to say that Texture2D.LoadImage() is very slow use WWW.loadTexture, but i can't use that as i am not uploading the photo from the client on a URL(webserver)

4)> Used Texture2D.LoadRawTexture , has the same effect poor performance.

5)> They said use threads , but i have never used them before and will they work on Android device? and how to use them in this case ?

6)> Please consider that i am a Bachelor Student of Computer Science(Studying currently in 5th semester) and have no practical experience, and this is my first time working on a networking project, so i can't afford any plugins, and please whatever you explain just remember that i am a student and not a professional.

Thankyou for your time!. Much obliged!

Comment
Add comment · Show 1
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image syalanurag1991 · Mar 23, 2018 at 03:34 AM 0
Share

Hey, have you solved this problem? I am doing a similar project.

2 Replies

· Add your reply
  • Sort: 
avatar image
0
Best Answer

Answer by Brain-_-Fail · Mar 23, 2018 at 06:46 AM

I found the solution to my question, the old unity legacy network system had flaws and didn't provide much control over what you do.I switched to UNET(The new networking architecture in unity), tweaked some parameters and i was done.A fact to note is that some cameras can't do good framerates in low lighting conditions, due to exposure timings, this problem cannot be solved with software tweaking of any sort except if you can tweak the hardware driver.Here are some of the important parameters you should tweak in UNET host topology definition.Below is what i've setup for my purposes.




     NetworkTransport.Init();
     cc = new ConnectionConfig();

     reliableChannel = cc.AddChannel(QosType.Reliable);
     unreliableChannel = cc.AddChannel(QosType.Unreliable);
     fragmentedReliableChannel = cc.AddChannel(QosType.ReliableFragmented);
     fragmentedUnreliableChannel = cc.AddChannel(QosType.UnreliableFragmented);

     cc.PacketSize = 1470;
     cc.MinUpdateTimeout = 1; 
     cc.SendDelay = 10;
     cc.BandwidthPeakFactor = 100;
     cc.InitialBandwidth = 1536000;        // this is the most important parameter
     cc.FragmentSize = 220;
     cc.MaxSentMessageQueueSize = 250;
Comment
Add comment · Show 8 · Share
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image syalanurag1991 · Mar 23, 2018 at 07:00 AM 0
Share

Thanks for your reply @Xy-aaBLa.

I have been following this thread till now: https://forum.unity.com/threads/stream-video-through-network.464693/

But I am not able to get a complete picture about where to start with. I am doing this for my school project. Please tell me something, about how I can proceed.

I want to have 2 apps, both made in Unity. One app is a client and another is a server. Both of them run on different machines on the same local network. The server is the one which transmits the webcam video.

Also, where does the above code go?

avatar image Brain-_-Fail syalanurag1991 · Mar 23, 2018 at 03:08 PM 0
Share

Here start with this video

https://www.youtube.com/watch?v=qGkkaNkq8co

avatar image syalanurag1991 Brain-_-Fail · Mar 24, 2018 at 03:36 AM 0
Share

Thanks a ton. I have been looking for this.

But just one quick question:

Have you tested the app on two separate physical machines?

I have my app ready. I made my laptop a sender and my desktop a receiver, but they don't seem to connect somehow.

Show more comments
avatar image rajanerve · Mar 26, 2018 at 11:49 AM 0
Share

Hello. I am trying to do the same. videostrea$$anonymous$$g from unity pc to android phone using RPC calls but i am trying to send the renderTexture. If u have any ideas on how i can achieve this please help. your help will be very much appreciated.

avatar image Brain-_-Fail rajanerve · Apr 30, 2018 at 11:06 AM 0
Share

I don't understand why you need RenderTexture for that, but to answer your question, you can't directly send RenderTexture over network without first converting it into Texture2D and then to Png or Jpg bytes buffer.You'll have to convert the RenderTexture to Texture2D and then convert the Texture2D to png or jpg bytes and then send the bytes over network.You can convert RenderTexture to Texture2D like this

       RenderTexture renTexture;
       Texture2D texture = new Texture2D(512, 512, TextureFormat.RGB24, false);


       // Convert the RenderTexture to Texture2D

       RenderTexture.active = renTexture;
       texture.ReadPixels(new Rect(0, 0, renTexture.width, renTexture.height), 0, 0);
       texture.Apply();

       // Now convert the Texture2D to bytes

       byte[] bytes = texture.EncodeToPNG();

       // Now you can send this over a network......



avatar image dragadaga · Jun 30, 2018 at 07:43 AM 0
Share

Hello! what is the delay and how many frames per second do you get? thank you!

avatar image Brain-_-Fail dragadaga · May 22, 2020 at 05:31 AM 0
Share

The delay is not too much. I remember I used to get 30fps If the sender of the stream was a mobile phone and 60fps if the sender was a PC. In low light conditions the frame rate can drop significantly as it depends on the camera hardware and the exposure ti$$anonymous$$gs it provides.

avatar image
1

Answer by thelghome · May 22, 2019 at 02:42 PM

FMETP STREAM is an All-in-One Game View Streaming Solution on almost all platforms.

Tested on: iOS/Android/Mac/PC/VR/AR/WebGL/HTML/Linux/HoloLens/MagicLeap..etc

Asset Store: https://assetstore.unity.com/packages/slug/143080

PS: All Source Code written in C#, free to modify for specific case.

Comment
Add comment · Share
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Your answer

Hint: You can notify a user about this post by typing @username

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this Question

Answers Answers and Comments

6 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

How o I prevent streamed video stutter? 1 Answer

Is 4K webcam compatible with Unity 5.6 video player/webcamtexture? 0 Answers

Decoding a video on the GPU 0 Answers

How to display a live streaming video in Unity? 0 Answers

Pixel Stream - Camera streaming 1 Answer


Enterprise
Social Q&A

Social
Subscribe on YouTube social-youtube Follow on LinkedIn social-linkedin Follow on Twitter social-twitter Follow on Facebook social-facebook Follow on Instagram social-instagram

Footer

  • Purchase
    • Products
    • Subscription
    • Asset Store
    • Unity Gear
    • Resellers
  • Education
    • Students
    • Educators
    • Certification
    • Learn
    • Center of Excellence
  • Download
    • Unity
    • Beta Program
  • Unity Labs
    • Labs
    • Publications
  • Resources
    • Learn platform
    • Community
    • Documentation
    • Unity QA
    • FAQ
    • Services Status
    • Connect
  • About Unity
    • About Us
    • Blog
    • Events
    • Careers
    • Contact
    • Press
    • Partners
    • Affiliates
    • Security
Copyright © 2020 Unity Technologies
  • Legal
  • Privacy Policy
  • Cookies
  • Do Not Sell My Personal Information
  • Cookies Settings
"Unity", Unity logos, and other Unity trademarks are trademarks or registered trademarks of Unity Technologies or its affiliates in the U.S. and elsewhere (more info here). Other names or brands are trademarks of their respective owners.
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Default
  • Help Room
  • META
  • Moderators
  • Explore
  • Topics
  • Questions
  • Users
  • Badges