koobas.hobune.stream
How to use Graphics.Blit with custom input textures
I am facing some problems in converting an image from YUV to RGB format.
I am using the Google Tango API with Unity. It gives me the bytes of an image in YUV format in the following callback function:
OnTangoImageAvailableEventHandler(TangoEnums.TangoCameraId cameraId, TangoUnityImageData imageBuffer)
Fortunately, they already have a YUV2RGB shader together with a material to convert images in their YUV format to RGB. So my plan was to create the two textures that the shader expects as input from the imageBuffer.data byte array. I then set those to the material from Tango and use Graphics.Blit to render the result to a RenderTexture. This is the function that I use for this task:
public void OnTangoImageAvailableEventHandler(TangoEnums.TangoCameraId cameraId, TangoUnityImageData imageBuffer)
{
// Split buffer into y and uv part
byte[] yDataPart = new byte[imageBuffer.width * imageBuffer.height];
Array.Copy (imageBuffer.data, 0, yDataPart, 0, imageBuffer.width * imageBuffer.height);
byte[] uDataPart = new byte[2 * (imageBuffer.width / 2) * (imageBuffer.height / 2)];
Array.Copy (imageBuffer.data, imageBuffer.width * imageBuffer.height, uDataPart, 0, 2 * (imageBuffer.width / 2) * (imageBuffer.height / 2));
// Create the textures from the data
Texture2D yTex = new Texture2D(width / 4, height, TextureFormat.RGBA32, false);
yTex.LoadRawTextureData (yDataPart);
yTex.Apply ();
Texture2D uTex = new Texture2D((width / 2) / 4 * 2, height / 2, TextureFormat.RGBA32, false);;
uTex.LoadRawTextureData (uDataPart);
uTex.Apply ();
// Set them to shader
yuv2rgbMat.SetTexture ("_YTex", yTex);
yuv2rgbMat.SetTexture ("_UTex", uTex);
// Draw
Graphics.Blit (dummyRenderTex, rgbRenderTex, yuv2rgbMat);
}
In that code snippet, the yuv2rgbMat is set to the ARScreen material from tango, which has the YUV2RGB shader connected.
This leads directly to my first question:
Is it necessary to provide an input render texture to Graphics.Blit if I already set the input textures directly to the material?
Now besides from that question, here is the real problem. When I later what to create a Texture2D from the RenderTexture to work further with it, the texture is completely set to 0. This is the code I use to access the resulting texture:
public IEnumerator drawRGBTexture()
{
while (true) {
yield return new WaitForEndOfFrame ();
if (rgbRenderTex != null) {
RenderTexture.active = rgbRenderTex;
rgbTex.ReadPixels (new Rect (0, 0, width, height), 0, 0);
rgbTex.Apply ();
// At this point rgbTex is completely filled with zeros!!
// Even if there were a lot of non zero entries in the original YUV byte stream
RenderTexture.active = null;
}
}
}
This leads to my second question:
Why is the resulting texture filled with zeros and not with the correct output of the shader?
I made some progress in the problem, the conversion using Graphics.Blit works now. There are still problems with accessing the raw image data. Here is the link to a new question I posted on Stackoverflow: http://stackoverflow.com/questions/41730124/how-to-access-raw-data-from-rendertexture-in-unity
Hi,
Where does the OnTangoImageAvailableEventHandler function go. Did you edit and put it into one of the existing Tango .cs file? And, how do you make the Shader output to your UI Raw Image that has the (http://stackoverflow.com/questions/41730124/how-to-access-raw-data-from-rendertexture-in-unity) code to it?
Thanks, Che
Oh sorry I forgot to post that i solved the Problem. I had some uninutialized variables, that caused the error in consequence.
Hi,I'm developing an app with Tango SD$$anonymous$$(Farandole Unity 5),but I met a problem similar to your's.The core code is listed down:
private int tWidth = 0;
private int tHeight = 0;
private Texture2D dummyInput;
private Texture2D yTex;
private Texture2D uvTex;
private byte[] yDataPart;
private byte[] uvDataPart;
private RenderTexture mRenderTexture;
private Texture2D mTexture;
private bool mTextureReady = false;
public $$anonymous$$aterial YUV2RGB;
public GameObject mQuad;
public void OnTangoImageAvailableEventHandler(TangoEnums.TangoCameraId cameraID, TangoUnityImageData imageData){
if (TangoEnums.TangoCameraId.TANGO_CA$$anonymous$$ERA_COLOR == cameraID) {
if (!mTextureReady) {
tWidth = (int)imageData.width;
tHeight = (int)imageData.height;
dummyInput = new Texture2D (tWidth, tHeight, TextureFormat.RGBA32, false);
yDataPart = new byte[tWidth * tHeight];
uvDataPart = new byte[2 * (tWidth / 2) * (tHeight / 2)];
yTex = new Texture2D (tWidth / 4, tHeight, TextureFormat.RGBA32, false);
yTex.Apply ();
uvTex = new Texture2D ((tWidth / 2) / 4 * 2, tHeight / 2, TextureFormat.RGBA32, false);
uvTex.Apply ();
mRenderTexture = new RenderTexture (tWidth, tHeight, 0, RenderTextureFormat.ARGB32);
mTexture = new Texture2D (tWidth, tHeight, TextureFormat.ARGB32, false);
mTexture.Apply ();
mQuad.GetComponent<Renderer> ().material.mainTexture = mTexture;
mTextureReady = true;
StartCoroutine (UpdateTexture2D ());
}
Array.Copy (imageData.data, 0, yDataPart, 0, tWidth * tHeight);
Array.Copy (imageData.data, tWidth * tHeight, uvDataPart, 0, 2 * (tWidth / 2) * (tHeight / 2));
yTex.LoadRawTextureData (yDataPart);
yTex.Apply ();
uvTex.LoadRawTextureData (uvDataPart);
uvTex.Apply ();
YUV2RGB.SetFloat ("_TexWidth", tWidth);
YUV2RGB.SetFloat ("_TexHeight", tHeight);
YUV2RGB.SetTexture ("_YTex", yTex);
YUV2RGB.SetTexture ("_UTex", uvTex);
Graphics.Blit (dummyInput, mRenderTexture, YUV2RGB);
}
}
IEnumerator UpdateTexture2D(){
while (mTextureReady) {
yield return new WaitForEndOfFrame ();
RenderTexture.active = mRenderTexture;
mTexture.ReadPixels (new Rect (0, 0, tWidth, tHeight), 0, 0);
mTexture.Apply ();
RenderTexture.active = null;
}
}
But the output texture is just confusing,it's quite different from what I expected.Could you please tell me what's wrong with my code or is there any tutorial about it?
PS: The CPU method works fine.I mean,the imageData is just right,and if I process it directly to the texture2D(without using shader and renderTexture),the output is fine(what I expected).
Thank you.
Well,I fixed it partly.The fixed code is list below,it works fine in my app.
private int tWidth = 0;
private int tHeight = 0;
private Texture2D dummyInput;
private Texture2D yTex;
private Texture2D uvTex;
private byte[] yDataPart;
private byte[] uvDataPart;
private RenderTexture mRenderTexture;
private Texture2D mTexture;
private bool mTextureReady = false;
public $$anonymous$$aterial YUV2RGB;//$$anonymous$$aterial: TangoPrefabs/$$anonymous$$aterials/ar_screen
public GameObject mQuad;
public void OnTangoImageAvailableEventHandler(TangoEnums.TangoCameraId cameraID, TangoUnityImageData imageData){
if (TangoEnums.TangoCameraId.TANGO_CA$$anonymous$$ERA_COLOR == cameraID) {
CheckSize ((int)imageData.width, (int)imageData.height);
Array.Copy (imageData.data, 0, yDataPart, 0, tWidth * tHeight);
Array.Copy (imageData.data, tWidth * tHeight, uvDataPart, 0, 2 * (tWidth / 2) * (tHeight / 2));
yTex.LoadRawTextureData (yDataPart);
yTex.Apply ();
uvTex.LoadRawTextureData (uvDataPart);
uvTex.Apply ();
YUV2RGB.SetFloat ("_TexWidth", tWidth);
YUV2RGB.SetFloat ("_TexHeight", tHeight);
YUV2RGB.SetTexture ("_YTex", yTex);
YUV2RGB.SetTexture ("_UTex", uvTex);
mRenderTexture.DiscardContents ();
Graphics.Blit (dummyInput, mRenderTexture, YUV2RGB);
//It seems that only right-top region of mRenderTexture is rendered.Could someone tell me why?
GL.InvalidateState ();
RenderTexture.active = mRenderTexture;
mTexture.ReadPixels (new Rect (tWidth / 2, tHeight / 2, tWidth, tHeight), 0, 0);
mTexture.Apply ();
RenderTexture.active = null;
}
}
public void CheckSize(int _width, int _height){
if (!mTextureReady) {
tWidth = _width;
tHeight = _height;
dummyInput = new Texture2D (64, 64);
yDataPart = new byte[tWidth * tHeight];
uvDataPart = new byte[2 * (tWidth / 2) * (tHeight / 2)];
yTex = new Texture2D (tWidth / 4, tHeight, TextureFormat.RGBA32, false);
yTex.Apply ();
uvTex = new Texture2D ((tWidth / 2) / 4 * 2, tHeight / 2, TextureFormat.RGBA32, false);
uvTex.Apply ();
mRenderTexture = new RenderTexture (tWidth, tHeight, 0);
mTexture = new Texture2D (tWidth / 2, tHeight / 2);
mTexture.Apply ();
mQuad.SetActive (true);
mQuad.GetComponent<Renderer> ().material.mainTexture = mTexture;
mTextureReady = true;
}
if (tWidth == _width && tHeight == _height) {
return;
}
tWidth = _width;
tHeight = _height;
yDataPart = new byte[tWidth * tHeight];
uvDataPart = new byte[2 * (tWidth / 2) * (tHeight / 2)];
GC.Collect ();
DestroyImmediate (yTex);
DestroyImmediate (uvTex);
yTex = new Texture2D (tWidth / 4, tHeight, TextureFormat.RGBA32, false);
yTex.Apply ();
uvTex = new Texture2D ((tWidth / 2) / 4 * 2, tHeight / 2, TextureFormat.RGBA32, false);
uvTex.Apply ();
mRenderTexture.Release ();
mRenderTexture = new RenderTexture (tWidth, tHeight, 0);
DestroyImmediate (mTexture);
mTexture = new Texture2D (tWidth / 2, tHeight / 2);
mTexture.Apply ();
mQuad.GetComponent<Renderer> ().material.mainTexture = mTexture;
}
Thank you for your effort. But when I use your two guys's code in my project. What I can get is just a rectangular with light red color? Is there any special setting? @karyuu_s @hendrikgruss
