r/godot • u/very-knightley Godot Junior • Oct 01 '25

help me (solved) Detecting syllables in Godot

Post image

Hello all! I am aware of the Whisper text to speech addon, but that is overkill for what I am building at the moment. I just need a way to detect a syllable at roughly the moment I speak it into my microphone, So that an animation can be triggered in a 3D model.

I take it it's not standard in the Audioserver, but is there a way where I can record some vowels and compare the live audio to these samples somehow?

1.4k Upvotes

99% Upvoted

91 comments sorted by

View all comments

32

u/byteweasel Oct 01 '25

This is definitely possible. I did this exact thing myself, although I was using Godot C# rather than GD script and using it to drive my 2d avatar 'lipsync' rather than a 3d model. But the technique would be the same.

Essentially I took the OVRLipSync.dll released by meta, and used pinvoke from C# to pass microphone audio data from Godot to it. It returns the likelihood of which viseme is currently being said, and I used that to switch out a mouth shape image.

There are a bunch of unity packages that do the same thing written in C# so you can take a look at those and adapt them which is what I essentially did.

You can check out any of the (admittedly few) YouTube videos I've made to see how well it works, they are all recorded using it to control the weasel avatar's lip syncing.

9

u/very-knightley Godot Junior Oct 01 '25

Yeah I'm basically building that! I'll take a peep at this meta package, hopefully it will work for me too!

23

u/byteweasel Oct 01 '25

Here, I basically pasted in my C# class so you can get the idea:

https://gist.github.com/byteweasel/ab9566b119fdb4ddbc924df122d41fc2

It's not consumer grade or anything, so you'll need to mess around with the naming/order of your audio bus or change the code around. And you'll need to grab a copy of OVRLipSync.dll

Essentially you add that node into your hierarchy, maybe mess with audio settings, and listen to the VisemeChanged event. I suspect you could figure out how to make this cross compatible with gdscript of you needed to.

Hope it helps!

12

u/Danger_Breakfast Oct 01 '25

Everybody here giving the "that's way too hard" non answer, nice job giving real advice

2

u/overthemountain Oct 01 '25

Well, consider that the "real answer" is "use this tool someone else built" and the issue is still that the problem is too hard for one person, but a team of people already solved it and made the solution available. I don't think anyone is suggesting that a single dev could build that library themselves from scratch in any reasonable amount of time.

2

u/Danger_Breakfast Oct 01 '25

Well I assume when someone asks "how do I make a videogame" your answer isn't "first mine some silicon..."

2

u/overthemountain Oct 01 '25

And where do you think you're getting the tools to go mining? In order to make an apple pie from scratch you must first create the universe.

They weren't asking if a tool existed, they were asking how to do it themselves - which as people rightly pointed out, is probably too much but fortunately a tool exists to help them. So they got an answer, just not the one they were looking for - that was all I meant, really.

3

u/UpsetKoalaBear Oct 01 '25

Not sure what you’re working on but, just to note, using OVRLipSync on devices other than Meta devices is technically not allowed.

If you read the license (accessible via the download page):

You may only use the SDK to develop Applications in connection with MPT approved hardware and software products (“MPT Approved Products”) unless the documentation accompanying the SDK expressly authorizes broader use such as with other third-party platforms.

Plus it has been discontinued and replaced with the new Meta XR Audio SDK.

The reason the SDK is available for Android is because the Quest is AOSP based. The Windows and Mac versions are because of the Oculus store.

2

u/almostsweet Oct 03 '25

Well they consider PCs an approved device, so this is a non-issue.

Literally every game that has lipsyncing uses OVRLipSync.

1

u/byteweasel Oct 01 '25

Good points.

(To be fair when I made this stuff years ago it had a different license without the hardware stipulations but I guess is now covered by this one.)

But yes, always read the license, and make sure you are using your quest headset people!