koobas.hobune.stream
Stream live camera view over UDP
I'm trying to fly my drone over my network using UDP for the video out and TCP for the commands in. For now, I want to set up a simulator of sorts in Unity. Do any of you know how I could encode and stream a camera view over UDP?
Also, while I'm at it, if you know how to send data into Unity over TCP, lmk please!
I implemented similar system for my robots. From my experience it is not easy to do it efficiently in Unity. All the efficient solutions that I implemented were rather complex and involved native code.
One way to do it:
Encode to H.264 or other standard with FFmpeg or gstreamer
Send to the receiving station with UDP.
In Unity NativePlugin decode the video with FFmpeg or gstreamer (thread)
From Unity grab the video frame data pointer and fill texture(s) from it (thread synchronization)
Display with shader if necessary (for planar formats like YUV or NV12)
You may find example for:
Realsense D435
hardware encoding with Intel VAAPI (on Linux)
simple custom UDP protocol
hardware decoding (VAAPI and other technologies, on Linux)
display to UI Raw Image or to 3D scene texture
on my github repository: https://github.com/bmegli/hardware-video-streaming
The system has measured 100 ms order glass-to-glass latency over WiFi.
If I knew simpler way to do it I would happily switch to it.
This is still not perfect. I am afraid it makes a managed copy of memory when filling textures with Unity LoadRawTextureData(IntPtr data, ...)
You could also fill the texture on NativePlugin side (full control of memory, no excess copies). The downside is that you have to implement this for every backend (OpenGL, DirectX, etc) yourself. You also have to synchronize it with GL.IssuePluginEvent which as I understand will introduce 1 frame latency.
FMETP STREAM| Forum Could be one of your options, without touching the native scripts in Unity3D.
It's a cross-platform solution for live streaming.
