- Home /
How to synchronize native plugin texture update with orientation change from script
Hi,
I have a program which retrieves a 360-degrees panoramic image from a camera, as well as an orientation quaternion from a orientation-sensor mounted to that camera. The information is retrieved in real-time and written to shared memory. In Unity I have a sphere with inverted normals which acts as skydome, and a camera in the center of the sphere. I've created a native C++ DLL plugin and a C# script. The plugin reads the orientation quaternion and the image from shared memory and updates the texture, while the script assigns the orientation values to the skydome Transform component. My solution is similar to the example project described on the native plug-in interface page.
The problem is that I observed that the point of time where the texture content changes is not in sync with the point of time where the orientation of the skydome-sphere is applied. This leads to small jitter-noise, which is, unfortunately, very apparent to the viewer. Please take a look at this short video demonstration (password = jitter) of the problem. By replaying the image/sensor data more slowly and taking a slow-motion video of my computer screen, I observed that first the texture is updated (while some of the orientation is applied), and then in the next frame the rest of the orientation-change is applied
I'd like to mention that the orientation sensor data comes it at ~60 Hz, while the image data comes in at 28-32 Hz. My approach is to update the skydome texture whenever there is a new camera image available, but to only update the skydome orientation at these very same moments, not at the 60Hz it could provide. Were I to update orientation readings at 60 Hz, I would get jitter for sure (sampling problem). I'd also like to note that I've taken care of the different sensor delays (the difference between delays of < something happens in real world > to < camera image reflects this change >, or < something happens in real world > to < orientation sensor readings reflect this change >), by artificially delaying the data of faster of the two sensors.
Below you see an extract of my code:
[DllImport ("UnityPlugin")]
private static extern void SetSkydomeTextureFromUnity(System.IntPtr texture);
[DllImport("UnityPlugin")]
private static extern bool isNewCameraImageAvailable();
private bool updateOrientation;
IEnumerator Start() {
updateData(); //copy sensor/image content from shared memory to memory of the C++ plugin
//Create the skydome texture
Texture2D tex = new Texture2D(1920, 1080, TextureFormat.ARGB32, false);
tex.Apply(); //actually uploaded to the GPU
GameObject sphere = GameObject.Find("pSphere1"); //skydome mesh
sphere.renderer.material.mainTexture = tex;
SetSkydomeTextureFromUnity(tex.GetNativeTexturePtr()); //pass texture pointer to the plugin
yield return StartCoroutine("CallPluginAtEndOfFrames");
}
private IEnumerator CallPluginAtEndOfFrames ()
{
while (true) {
//Do work inbetween frames
yield return new WaitForEndOfFrame();
updateData();
//now apply the values to the different gameobjects
updateHeadTrackerOrientation(); //no more details necessary
updateOrientation = isNewCameraImageAvailable();
updateSkydomeOrientation(); //simply gets w, x, y, z from native plugin and assigns values to skydome transform object, but only if updateOrientation == true
//Update skydome Texture from C++ code immediately in case there is a new image available
if (updateOrientation) {
GL.IssuePluginEvent(1); //note: the parameter is an unused ID (ignored)
}
}
}
//note: nothing relevant in Update() or LateUpdate()
The update of the skydome texture itself takes place in C++ code in EXPORT_API void UnityRenderEvent(int eventID)
.
It seems to me that the maybe the rendering thread (which steps into EXPORT_API void UnityRenderEvent(int eventID)
) and the thread that executes the CallPluginAtEndOfFrames()
co-routine (updating the skydome orientation) are not properly synchronized. However, I don't know how to achieve this synchronization... :( Any help would be appreciated!
Best regards!
Your answer

Follow this Question
Related Questions
Are calls to C++ DLL handled asynchronously? 1 Answer
functional multithreaded c++ code crashes if executed in Unity as native plugin 1 Answer
Cannot make native plugin working on OS X. 0 Answers
How to create a native OS X plugin from a static C++ .a library 0 Answers
Using CVOpenGLTextureCache in Plugin on OSX - Get correct OpenGL context? 0 Answers