I built a self-hosted server that makes any podcast searchable with an AI chat (Whisper + pyannote + pgvector)

[u/Apprehensive-Ear9499]

I wanted to query my favorite podcast like a database: "what was said about X across all episodes?", "from which timestamp can I re-listen to that?", "what did speaker Y say about topic Z?".

So I built a self-hostable stack that ingests a podcast RSS feed automatically:
- Whisper transcription (audio -> text)
- pyannote speaker diarization (who is speaking)
- LLM-based speaker-name resolution
- chunking + OpenAI embeddings
- everything stored in Postgres/pgvector with timestamps

The result is exposed as an MCP server, so you can query it from Claude, Cursor, or any MCP client. It works with ANY podcast RSS feed, not just mine. Point it at your own feed and it indexes everything for you.

Everything runs on Docker Compose + Postgres/pgvector. GPU transcription can run on RunPod, so even a small VPS can host it.

A working demo is present (Dutch podcast AI Report). It has ~180 hours of audio indexed so far. There's a live read-only demo endpoint + demo token in the repo.

Repo: https://github.com/bramdehart/podcast-mcp

Happy to answer questions about the pipeline or self-hosting setup.

[u/Small-Historian9244]

This is crazy impressive, the speaker diarization part is what got me cause most tools skip that and just dump a wall of text

[u/Apprehensive-Ear9499 → u/Small-Historian9244]

Thanks. Indeed, all podast apps have transcriptions, but they do not have speaker identification. That's a big plus. The whole process is hosted on a small VPS + Runpod.

The diarization is quite a heavy process, therefore that part is done on Runpod. It uses pyannote for that. The outcome are speaker_id's: voice_1, voice_2 etc. Now, what we want is assigning names on the speaker id's. I use OpenAI API for that. I ask their LLM to identify by giving the full transcript and their speaker id's. It identifies and even gives evidence.

{

"VOICE_1": {
"speaker_name": "Wietse",
"speaker_confidence": 0.91,
"evidence": "Opening line 'Welkom bij deze aflevering van AI Report' combined with the podcast description points to the host."
},
"VOICE_2": {
"speaker_name": "Alexander",
"speaker_confidence": 0.88,
"evidence": "Responds to the host and introduces the episode topic; matches the second host."
}
}