Any way of "automatic" lip syncing?

Great stuff aldonaletto, I was looking for exactly this! However the BandVol function doesn't seem to respond to audio played with the PlayOneShot, only clips directly assigned to the audio component. Any idea how to get around that?

In the spirit of sharing, I managed to get ok looking results by using the value returned by your script and hooking them up to an additive animation.

I basically made a mouth animation which starts closed, opens with an O-like shape, then goes on to a wider shout style pose. This animation is enabled from the start with a maximum weight and zero playback speed. Since it's additive it won't affect any objects at frame 0. I then used your BandVol function to control the normalized time of the additive animation. Since the additive animation has nonlinear movement and some variation in it, it gave a more organic result than if I were to rotate the jaw or maybe fade a pose in and out by controlling its weight.

I also used a cutoff value making the character close his mouth at low values. Encouraging a more "talky" motion as opposed to a half-open vibrating pose that can happen at lower volumes. And finally a Lerp so I could tweak how smooth the mouth movements should be. In the end it worked well for my cartoony flappy-mouth character.

The extra variables used:

 private float mouthCurrentPose;
 private float mouthTargetPose;
 public float voiceVolumeCutoff;
 public float mouthBlendSpeed;

The setup of the additive animation from Start()

 animation["anim_talk"].layer = 5;
 animation["anim_talk"].blendMode = AnimationBlendMode.Additive;
 animation["anim_talk"].speed = 0.0f;
 animation["anim_talk"].weight = 1.0f;
 animation["anim_talk"].enabled = true;
 animation["anim_talk"].wrapMode = WrapMode.ClampForever;

and the function running the mouthpose

    void LipSynch()
 {
     mouthTargetPose = BandVol(frqLow,frqHigh)* volume;
 
 // Tweak the voiceVolumeCutoff to get a good result, I used 0.1f myself
     if(mouthTargetPose<voiceVolumeCutoff)
         mouthTargetPose = 0.0f;
     
     mouthCurrentPose = Mathf.Lerp(mouthCurrentPose,mouthTargetPose,Time.deltaTime*mouthBlendSpeed);
 
 // I didn't bother with clamping the result since the addditive animation clamps itself.
 // Tweak the volume value to get results between 0.0 and 1.0 from your voice samples.
     animation["anim_talk"].normalizedTime = mouthCurrentPose;
 }

You can get better results if you tweak the volume and voiceVolumeCutoff to match each voiceclip.

Here's what I've found:

1. Forum topic (discusses some alternatives)

2. Script on the wiki (if you model the individual sounds as separate meshes, it looks like this allows you to smoothly transition from one state to the other)

If you don't particularly care about the accuracy of the animations, just rig your characters' mouths and play an animation any time they're supposed to speak.

Alternatively, bones are just transforms so you should be able to reference them like any other transform. Just have your script adjust the magnitude of the movement based on the audio clip's volume. See Al's note below.

if I were doing lipsyncs, I would probably rig up some standard phoneme mouth shapes/blends (M-E-O-W), and write some sort of custom blending system that would interpret a magpie (or similar) script and blend to the appropriate mouth shape.

magpie, and other software like it, will take recorded dialogue and attempt to generate a 'timing script' based on phonemes it detects. My experience with it usually needed very little cleanup work- after that you have a pretty usable 'script' from which you can get all the timing info about your dialogue. From there you could easily write something that parsed the lipsync timing, and if a particular phoneme is detected, just blend to that shape.

This is all just theory- I haven't implemented it before, but if I were that's probably how I would do it if accuracy were any concern.

Hi,

I'm trying the here mentioned answers directly for the microphone input instead of a loaded audioclip and it doesn't work :( Is there any reason for that?

I'm using these functions (adapted from the ones mentioned here and in another post):

 private float GetVolume()
 {
     if(audio==null)
         return 0;
     float[] data = new float[samples];
     audio.GetOutputData(data, 0);
     
     //take the median of the recorded samples
     ArrayList s = new ArrayList();
     foreach (float f in data)
     {
         s.Add(Mathf.Abs(f));
     }
     s.Sort();
     return (float)s[samples / 2];
 }
 
 float fMax = 24000;
 private float HumanFreq(float fLow, float fHigh)
 {
     if(audio==null)
         return 0;
     float[] data = new float[samples];
     fLow = Mathf.Clamp(fLow, 20, fMax); // limit low...
     fHigh = Mathf.Clamp(fHigh, fLow, fMax); // and high frequencies
     // get spectrum: freqData[n] = vol of frequency n * fMax / nSamples
     audio.GetSpectrumData(data, 0, FFTWindow.BlackmanHarris); 
     int n1 = (int)Mathf.Floor(fLow * samples / fMax);
     int n2 = (int)Mathf.Floor(fHigh * samples / fMax);
     float sum = 0;
     // average the volumes of frequencies fLow to fHigh
     for (var i=n1; i<=n2; i++){
         sum += data[i];
     }
     return sum / (n2 - n1 + 1);
 }

GetVolume virtually always returns 0 (with random exceptions), and HumanFreq does not filter wheather it's a human voice or just noise. Am I missing something? I tried changing samples value, but with no effect.