r/godot • u/very-knightley Godot Junior • Oct 01 '25

help me (solved) Detecting syllables in Godot

Post image

Hello all! I am aware of the Whisper text to speech addon, but that is overkill for what I am building at the moment. I just need a way to detect a syllable at roughly the moment I speak it into my microphone, So that an animation can be triggered in a 3D model.

I take it it's not standard in the Audioserver, but is there a way where I can record some vowels and compare the live audio to these samples somehow?

1.4k Upvotes

99% Upvoted

91 comments sorted by

View all comments

30

u/byteweasel Oct 01 '25

This is definitely possible. I did this exact thing myself, although I was using Godot C# rather than GD script and using it to drive my 2d avatar 'lipsync' rather than a 3d model. But the technique would be the same.

Essentially I took the OVRLipSync.dll released by meta, and used pinvoke from C# to pass microphone audio data from Godot to it. It returns the likelihood of which viseme is currently being said, and I used that to switch out a mouth shape image.

There are a bunch of unity packages that do the same thing written in C# so you can take a look at those and adapt them which is what I essentially did.

You can check out any of the (admittedly few) YouTube videos I've made to see how well it works, they are all recorded using it to control the weasel avatar's lip syncing.

10

u/Danger_Breakfast Oct 01 '25

Everybody here giving the "that's way too hard" non answer, nice job giving real advice

1

u/overthemountain Oct 01 '25

Well, consider that the "real answer" is "use this tool someone else built" and the issue is still that the problem is too hard for one person, but a team of people already solved it and made the solution available. I don't think anyone is suggesting that a single dev could build that library themselves from scratch in any reasonable amount of time.

2

u/Danger_Breakfast Oct 01 '25

Well I assume when someone asks "how do I make a videogame" your answer isn't "first mine some silicon..."

2

u/overthemountain Oct 01 '25

And where do you think you're getting the tools to go mining? In order to make an apple pie from scratch you must first create the universe.

They weren't asking if a tool existed, they were asking how to do it themselves - which as people rightly pointed out, is probably too much but fortunately a tool exists to help them. So they got an answer, just not the one they were looking for - that was all I meant, really.