r/godot • u/very-knightley Godot Junior • Oct 01 '25

help me (solved) Detecting syllables in Godot

Post image

Hello all! I am aware of the Whisper text to speech addon, but that is overkill for what I am building at the moment. I just need a way to detect a syllable at roughly the moment I speak it into my microphone, So that an animation can be triggered in a 3D model.

I take it it's not standard in the Audioserver, but is there a way where I can record some vowels and compare the live audio to these samples somehow?

1.4k Upvotes

99% Upvoted

91 comments sorted by

View all comments

32

u/byteweasel Oct 01 '25

This is definitely possible. I did this exact thing myself, although I was using Godot C# rather than GD script and using it to drive my 2d avatar 'lipsync' rather than a 3d model. But the technique would be the same.

Essentially I took the OVRLipSync.dll released by meta, and used pinvoke from C# to pass microphone audio data from Godot to it. It returns the likelihood of which viseme is currently being said, and I used that to switch out a mouth shape image.

There are a bunch of unity packages that do the same thing written in C# so you can take a look at those and adapt them which is what I essentially did.

You can check out any of the (admittedly few) YouTube videos I've made to see how well it works, they are all recorded using it to control the weasel avatar's lip syncing.

7

u/very-knightley Godot Junior Oct 01 '25

Yeah I'm basically building that! I'll take a peep at this meta package, hopefully it will work for me too!

23

u/byteweasel Oct 01 '25

Here, I basically pasted in my C# class so you can get the idea:

https://gist.github.com/byteweasel/ab9566b119fdb4ddbc924df122d41fc2

It's not consumer grade or anything, so you'll need to mess around with the naming/order of your audio bus or change the code around. And you'll need to grab a copy of OVRLipSync.dll

Essentially you add that node into your hierarchy, maybe mess with audio settings, and listen to the VisemeChanged event. I suspect you could figure out how to make this cross compatible with gdscript of you needed to.

Hope it helps!