Wayback Machinekoobas.hobune.stream
May JUN Jul
Previous capture 12 Next capture
2021 2022 2023
1 capture
12 Jun 22 - 12 Jun 22
sparklines
Close Help
  • Products
  • Solutions
  • Made with Unity
  • Learning
  • Support & Services
  • Community
  • Asset Store
  • Get Unity

UNITY ACCOUNT

You need a Unity Account to shop in the Online and Asset Stores, participate in the Unity Community and manage your license portfolio. Login Create account
  • Blog
  • Forums
  • Answers
  • Evangelists
  • User Groups
  • Beta Program
  • Advisory Panel

Navigation

  • Home
  • Products
  • Solutions
  • Made with Unity
  • Learning
  • Support & Services
  • Community
    • Blog
    • Forums
    • Answers
    • Evangelists
    • User Groups
    • Beta Program
    • Advisory Panel

Unity account

You need a Unity Account to shop in the Online and Asset Stores, participate in the Unity Community and manage your license portfolio. Login Create account

Language

  • Chinese
  • Spanish
  • Japanese
  • Korean
  • Portuguese
  • Ask a question
  • Spaces
    • Default
    • Help Room
    • META
    • Moderators
    • Topics
    • Questions
    • Users
    • Badges
  • Home /
  • Help Room /
avatar image
0
Question by loki-k-rasmussen · Jul 20, 2016 at 04:31 PM · networkingoptimizationtextureswebcamtexture

Optimize RPC streamed video as byte array (frame by frame jpg)

I'm working on networking a webcam over Photon so that I can send the camera view of one player over the network and project in on the screen in the environment of other players on the network. So far, the process is working and the image is appearing over the network, but the frame rate slowly decreases until eventually it crashes the network. I've done a lot of reading and it seems pretty obvious that this is because of how large the byte array being sent is and how slow the tex.LoadImage() and tex.EncodeToJPG functions are. Generally the byte array length is around 90,000.

So, I need the best way to optimize this. I've considered A) somehow using OpenGL to pass the byte array to a shader and doing the conversion in the shader, which would bypass the tex.LoadImage() function. Though, from what I've read this seems unlikely. I've done some shader programming but would need more information on the tex.EncodeToJPG() byte format

or B) and this is the more likely way I think find a way to write something similar to the algorithm mpeg uses, to only send frames when there's a change in the image frame, possibly even send only that changed data

C) whatever you can suggest

this is the code I'm using to send the byte array over the RPC

sing UnityEngine; using System.Collections; using System.Collections.Generic; using System.Runtime.InteropServices; using System; using System.IO; //using UnityEditor;

public class CamCaptureStream : Photon.MonoBehaviour {

 public Renderer renderer;
 public WebCamTexture webCamTexture;
 public string deviceName;
 Color32[] data;
 public Texture2D tex;
 byte[] newbytes;
 byte[] bytesToSend;
 public Texture2D MyTexture;
 int count;
 //static BytesCompression compress;



 void Start(){
     deviceName = WebCamTexture.devices[0].name;
     webCamTexture = new WebCamTexture(deviceName, 20, 20, 20);
     GetComponent<VoiceChat>().enabled = true;
     GetComponent<VoiceChat>().ready = true;
     GetComponent<AudioSource>().enabled = true;
     GetComponent<AudioListener>().enabled = true;
     count = 0;
     
     if(photonView.isMine){
         webCamTexture.Play();
     }
         
 }

 void Update(){
     if(webCamTexture.width < 100){
         Debug.Log("Waiting for Camera to Load");
         return;
     }
     else{
         data = new Color32[webCamTexture.width*webCamTexture.height];
         Debug.Log(data.Length);
         MyTexture = new Texture2D(webCamTexture.width, webCamTexture.height, TextureFormat.RGB24, false);
         tex = new Texture2D(webCamTexture.width, webCamTexture.height, TextureFormat.RGB24, false);
     }

     if(photonView.isMine){
         StartCoroutine(GetPixels()); 
         //UDPSend.sendString(bytesToSend);
         //Debug.Log("Fails after this: Probably too much bytes being sent");


     }
     else{
             tex.LoadImage(bytesToSend);
             renderer.material.mainTexture = tex;
             count = 0;
         
     }
         
     //renderer.material.SetTexture("tex", tex);
     //Destroy(tex);
 }

 [PunRPC]
 void CamerStream(byte[] data){
     tex.LoadImage(data);
     renderer.material.mainTexture = tex;
     tex.Apply();
 }

 private static byte[] Color32ArrayToByteArray(Color32[] colors)
 {
     byte[] bytes = new byte[colors.Length*4];
     for(int i = 0; i < bytes.Length/4; i+=4){
         bytes[i] = colors[i].r;
         bytes[i+1] = colors[i].g;
         bytes[i+2] = colors[i].b;
         bytes[i+3] = colors[i].a;
     }
     return bytes;
 }

 IEnumerator GetPixels(){
         MyTexture.SetPixels32(webCamTexture.GetPixels32(data));
         //MyTexture.Compress(false);
         //bytesToSend = MyTexture.GetRawTextureData();
         bytesToSend = MyTexture.EncodeToJPG();
         yield return bytesToSend;
         print("Bytes to Send Size: " + bytesToSend.Length);
         photonView.RPC("CamerStream", PhotonTargets.All, bytesToSend);
 
 }


 

}

Comment
Add comment · Show 1
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image einstyle · Jun 28, 2017 at 06:53 PM 0
Share

I'm trying to doing the same thing. Did you succeed?

0 Replies

· Add your reply
  • Sort: 

Your answer

Hint: You can notify a user about this post by typing @username

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this Question

Answers Answers and Comments

83 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

How to use mipmaps in code + detecting different faces of a mesh? 0 Answers

How to fix: transform.position assign attempt for 'Player(Clone)' is not valid. Input position is { NaN, NaN, NaN }. UnityEngine.Transform:Translate(Vector3) FloatingPlayer2DController:Update() (at Assets/FloatingPlayer2DController.cs:59) 2 Answers

Stream webcam via Multiplayer networking or websocket 2 Answers

Optimize the standard shaders texture slots? 0 Answers

Yield to see if NetworkServer sends back anything? 0 Answers


Enterprise
Social Q&A

Social
Subscribe on YouTube social-youtube Follow on LinkedIn social-linkedin Follow on Twitter social-twitter Follow on Facebook social-facebook Follow on Instagram social-instagram

Footer

  • Purchase
    • Products
    • Subscription
    • Asset Store
    • Unity Gear
    • Resellers
  • Education
    • Students
    • Educators
    • Certification
    • Learn
    • Center of Excellence
  • Download
    • Unity
    • Beta Program
  • Unity Labs
    • Labs
    • Publications
  • Resources
    • Learn platform
    • Community
    • Documentation
    • Unity QA
    • FAQ
    • Services Status
    • Connect
  • About Unity
    • About Us
    • Blog
    • Events
    • Careers
    • Contact
    • Press
    • Partners
    • Affiliates
    • Security
Copyright © 2020 Unity Technologies
  • Legal
  • Privacy Policy
  • Cookies
  • Do Not Sell My Personal Information
  • Cookies Settings
"Unity", Unity logos, and other Unity trademarks are trademarks or registered trademarks of Unity Technologies or its affiliates in the U.S. and elsewhere (more info here). Other names or brands are trademarks of their respective owners.
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Default
  • Help Room
  • META
  • Moderators
  • Explore
  • Topics
  • Questions
  • Users
  • Badges