koobas.hobune.stream
Live puppetry and-or facial expression capture-options
Currently exploring what options there are for doing LIVE capture of facial expressions within Unity. There are plugins for FaceShift, Brekel Kinect and also came across Mixamo recently acquired by Adobe, though not sure if the latter can actually drive LIVE animation or simply used for capture. Has anyone reading this explored this area and can recommend the pros and cons of various solutions?
Well I'll tell you right off the bat, that since it is real-time, it will not be as accurate in 2 ways:
It may not look realistic
There may be glitches
That's what happens when you have limited time to process what little data you have, though.
I'm not exactly sure how you would do facial tracking, but to somewhat give you an idea, most tracking is done with points from video. It uses a wide variety of points that are strategically placed at high-contrast points. So you take these points you're tracking, and simply apply the transformation to a rig in you scene. Now the tricky part is getting this camera work and sending that data into Unity for transformation data. I've tried something like this in the past with Cinema 4D and an Xbox $$anonymous$$inect, eventually getting it to work, but your best bet in the end is to get plugins.
Good luck.
Google Epcot turtle talk to get an idea of what I hope to achieve
