koobas.hobune.stream
Pass a RenderTexture to a plugin with zero copies
I have camera that renders to a RenderTexture. I would like to pass the contents of the RenderTexture to a native plugin. From the Unity's native plugin example, a native plugin can only handle ID3D11Texture2D pointers which correspond to Unity's Texture2D.GetNativeTexturePtr objects. Therefore, I have a functional version whereby the RenderTexture's output is first copied to a Texture2D with ReadPixels:
private IEnumerator DoEveryFrame()
{
//Wait for graphics to render
yield return new WaitForEndOfFrame();
// Remember current render textures
RenderTexture currentActiveRT = RenderTexture.active;
RenderTexture currentCamRT = cam.targetTexture;
// Force rendering of the camera to my render texture
cam.targetTexture = rt;
cam.Render(); // 2nd Render seems to be necessary, but why??
cam.targetTexture = currentCamRT;
// Get a copy of the rendered data
RenderTexture.active = rt;
screenShot.ReadPixels(new Rect(0, 0, Screen.width, Screen.height), 0, 0);
screenShot.Apply(); // hits perf significantly but needed otherwise actual copy does not occur
// Restorie previously assigned render texture
RenderTexture.active = currentActiveRT;
//Destroy(rt);
// Issue a plugin event with arbitrary integer identifier.
// The plugin can distinguish between different
// things it needs to do based on this ID.
// For our simple plugin, it does not matter which ID we pass here.
GL.IssuePluginEvent (200);
}
Unfortunately, this method of using ReadPixels() and then Apply() significantly hurts performance. I don't actually need the contents of RenderTexture on the CPU. How can I accomplish either of the following:
[Preferred] Pass a valid reference to the RenderTexture directly to the native plugin without any data copying. This would be ideal. However, using RenderTexture.GetNativeTexturePtr resulted in all-black texture as seen by native plugin.
Make a GPU-to-GPU copy of Render Texture to Texture2D. A copy is still involved, but at least it is not crossing the bus.
My second minor question is why do I need to perform a second Camera.Render()? Why doesn't this work:
RenterTexture.active = cam.targetTexture;
screenShot.ReadPixels(...);
screenShot.Apply();
This must be doable; I'm just missing the magic invocation... :-)
Is there anyone that can help with this? Seems like zero copy ought to be something that can be done...
Have you made any progress with this? It seems we're trying to do the exact same thing. $$anonymous$$y C# is practically identical to yours and I'm currently trying to write a C++ plugin to do #2 (RenderTexture to Texture2D). However, casting RenderTexture to ID3D11Texture2D crashes my plugin.
Yes, I was able to pass a rt.GetNativeTexturePtr directly and it works. However, it seems I still must invoke the ReadPixels and Apply calls:
// Get a copy of the rendered data
RenderTexture.active = rt;
screenShot.ReadPixels(new Rect(0, 0, Screen.width, Screen.height), 0, 0);
screenShot.Apply(); // hits perf significantly but needed otherwise actual copy does not occur
otherwise the rt does not actually get filled with the new data! This is very confusing and kills performance...
yangguo, did you figure this out? I'm looking at using GetNativeTexturePtr as well.
Did you try to switch to OpenGL (on Windows with -force-opengl). I can use the OpenGL texture in my native C++ plugin.
At the moment I am struggling with Android. Same code as on Windows but black texture as well (no OpenGL errors). Maybe the Direct3D and Android OpenGL ES 2.0 are somehow related? Did you find a solution yet?
Could you post some example code? $$anonymous$$aybe from the C++ plugin as well? I am also trying to get this working in OpenGL by passing RenderTexture.GetNativeTexturePtr, but have only seen a black screen so far.
If you want to grab screen contents I think you can do this:
RenterTexture.active = null;
screenShot.ReadPixels(...);
screenShot.Apply();
Because what you're doing right now is rendering to rt first and then grabbing, when you can grab from screen directly.
Paul: your solution works, but is too slow for me. I am trying to avoid ReadPixels() and Apply() since these transfer data to CPU. I want to keep all data on GPU and let the plugin work on the data in the GPU
What does your plugin do? I don't believe it's possible to work with data directly on the GPU except in a shader.
The goal of the plugin is to pass the render texture to $$anonymous$$edia Foundation for further processing.
