koobas.hobune.stream
How to change image at runtime using bytes
Hi,
I am using the new Unity UI Image component. I am also using a Kinect. What I want to do is write the bytes from the Kinect image, to the Image component in Unity.
Anyone any ideas on the best way to do this?
I've saw a few examples of loading an image from disk into an image component, but these wont be saved on disk.
Thanks in advance!!!!
Which Kinect are you using?
Kinect v2 (The one that came with the Xbox One) has a set of Unity plugins and samples.
Here's how they handle it:
(in Start)
_Reader = _Sensor.ColorFrameSource.OpenReader();
var frameDesc = _Sensor.ColorFrameSource.CreateFrameDescription(ColorImageFormat.Rgba);
_Texture = new Texture2D(frameDesc.Width, frameDesc.Height, TextureFormat.RGBA32, false);
_Data = new byte[frameDesc.BytesPerPixel * frameDesc.LengthInPixels];
(In Update)
var frame = _Reader.AcquireLatestFrame();
if (frame != null)
{
frame.CopyConvertedFrameDataToArray(_Data, ColorImageFormat.Rgba);
_Texture.LoadRawTextureData(_Data);
_Texture.Apply();
frame.Dispose();
frame = null;
}
In Start, you should be able to assign the Texture2D _Texture to a new Sprite like so:
myImage.sprite = Sprite.Create(_Texture, sourceRect, pivot, scaling);
and it will get updated as new frames become available from the sensor.
If you're using the original Kinect, the code for reading data from it will be a bit different, but the basic principle should still apply.
Problem is that CreateFrameDescription should change the FrameDescription of the ColorFrameSource.
Creates a FrameDescription object that describes a color frame with the provided format and resolution. This replaces the FrameDescription property found on other image streams.
($$anonymous$$SDN $$anonymous$$inect v2 API Overview)
But in reality, CreateFrameDescription seems to have a bug because while it does create a new FrameDescription object in memory, it doesn't associate it to the ColorFrameSource.
If this worked correctly then you could use frame.CopyRawFrameDataToArray(_Data) and avoid using ~4ms to ~9ms to convert color data to RGBA.
$$anonymous$$eep in $$anonymous$$d that the above comment mainly applies to those whose sensor's raw color data come in YUV2 or anything other than the format you need. BGRA is simple to switch to RGBA in a shader but YUV2 which is half in size would need more time to decompress into BGRA/RGBA.
Furthermore, if you consider performance, it might be wise to consider pinning the buffers in memory to avoid constant allocations for your byte arrays and using the Copy....FrameDataToIntPtr ins$$anonymous$$d.
