- Home /
Highlighting of the text according to the sound file(like karaoke)
Hello! I have a text and sound file(i mean the sound file, where it is recorded as someone read this text).
How do I implement highlighting of the text according to the voice acting?(like karaoke) It's for interactive book for children.
So we open the page of the book and hear the author read the text and see highlighting the right words.
Answer by clunk47 · Oct 07, 2013 at 03:36 PM
There's no total speech recognition built into Unity. All you can really do is mess around with the spectrum data, but that would be a long shot. The only thing you can do besides that is sync it all manually. That's with Unity alone. Another thing you could do is look for a Plugin, which you would only be able to use with Unity Pro or Mobile.
Just to help get started a bit, here's an example of how to use SpecData if you decide to go with it.
using UnityEngine;
using System.Collections;
public class specData : MonoBehaviour
{
public AudioListener Listener;
public int numSamples = 128;
public GameObject abar;
float[] numberleft;
float[] numberright;
GameObject[] thebarsleft;
GameObject[] thebarsright;
float spacing;
float width;
void Start ()
{
thebarsleft = new GameObject[numSamples];
thebarsright = new GameObject[numSamples];
spacing = .4f - (numSamples * 0.001f);
width = .1f - (numSamples * 0.001f);
for(int i=0; i < numSamples; i++)
{
float xpos = i*spacing -8.0f;
Vector3 positionleft = new Vector3(xpos,3, 0);
thebarsleft[i] = (GameObject)Instantiate(abar, positionleft, Quaternion.identity) as GameObject;
thebarsleft[i].transform.localScale = new Vector3(width,1,0.2f);
Vector3 positionright = new Vector3(xpos,-3, 0);
thebarsright[i] = (GameObject)Instantiate(abar, positionright, Quaternion.identity) as GameObject;
thebarsright[i].transform.localScale = new Vector3(width,1,0.2f);
}
}
void Update ()
{
float[] numberleft = AudioListener.GetSpectrumData (numSamples, 0,FFTWindow.BlackmanHarris);
float[] numberright = AudioListener.GetSpectrumData (numSamples, 1,FFTWindow.BlackmanHarris);
for(int i=0; i < numSamples; i++){
if (float.IsInfinity(numberleft[i]*30) || float.IsNaN(numberleft[i]*30)){
}else{
thebarsleft[i].transform.localScale = new Vector3(width, numberleft[i]*30,0.2f);
thebarsright[i].transform.localScale = new Vector3(width, numberright[i]*30,0.2f);
}
}
}
}
Thanks! And what are the ways to implement the highlighting letters, words?
For that, I'd just use GUI.DrawTexture to draw a highlight texture behind the Rect of each word. It will be pretty simple math to get the Rects right. I'm sure there are other ways, but this is how I know you could do it with the GUI class in Unity.
Thanks. Do you have any examples or links with you. please share. !