Stream live camera view over UDP
I'm trying to fly my drone over my network using UDP for the video out and TCP for the commands in. For now, I want to set up a simulator of sorts in Unity. Do any of you know how I could encode and stream a camera view over UDP?
Also, while I'm at it, if you know how to send data into Unity over TCP, lmk please!
Answer by bmegli · May 15, 2019 at 12:43 PM
I implemented similar system for my robots. From my experience it is not easy to do it efficiently in Unity. All the efficient solutions that I implemented were rather complex and involved native code.
One way to do it:
Encode to H.264 or other standard with FFmpeg or gstreamer
Send to the receiving station with UDP.
In Unity NativePlugin decode the video with FFmpeg or gstreamer (thread)
From Unity grab the video frame data pointer and fill texture(s) from it (thread synchronization)
Display with shader if necessary (for planar formats like YUV or NV12)
You may find example for:
Realsense D435
hardware encoding with Intel VAAPI (on Linux)
simple custom UDP protocol
hardware decoding (VAAPI and other technologies, on Linux)
display to UI Raw Image or to 3D scene texture
on my github repository: https://github.com/bmegli/hardware-video-streaming
The system has measured 100 ms order glass-to-glass latency over WiFi.
If I knew simpler way to do it I would happily switch to it.
This is still not perfect. I am afraid it makes a managed copy of memory when filling textures with Unity LoadRawTextureData(IntPtr data, ...)
You could also fill the texture on NativePlugin side (full control of memory, no excess copies). The downside is that you have to implement this for every backend (OpenGL, DirectX, etc) yourself. You also have to synchronize it with GL.IssuePluginEvent which as I understand will introduce 1 frame latency.
Answer by thelghome · Jul 02, 2019 at 02:21 PM
FMETP STREAM| Forum Could be one of your options, without touching the native scripts in Unity3D.
It's a cross-platform solution for live streaming.
Your answer
Follow this Question
Related Questions
How to show video of an IP camera 0 Answers
Streaming video camera input from WebCamTexture on mobile? (Possibly via WebRTC or websocket?) 1 Answer
Stream while simultaneously downloading video from URL using video player 0 Answers
Play and capture a video at the same time 0 Answers
SPACE SHOOTER SHOOTING SHOTS TUTORIAL HELP (UNITY 5 C#) 3 Answers