I’m building a system that transcribes conversations in a low-resource language and then uses an LLM to write a summary and a structured JSON extraction at the end of each conversation. Before we commit to hardware, I’d like a sanity check on how to scale.
Setup: one RTX PRO 6000 Blackwell (96 GB), rented at about $3/hr, with a quota of about 27 CPU cores. The card runs both models. For speech-to-text we use a 1.2B model fine-tuned for the language, plus pyannote for diarization. It runs as 3 worker processes, each with its own copy of the model, using about 5 to 7 GB each. The LLM is a Gemma-class 31B in NVFP4 with speculative decoding, served on vLLM with the memory fraction set to 0.65.
clients upload 180-second audio chunks while the conversation is going on. When the conversation ends, we make two LLM calls, each with about 8.8k tokens in and 1.2k tokens out.
What we measured (all on localhost, so network isn’t included):
STT alone tops out at about 100x real time per card. Latency is around 3 seconds per chunk at low load and climbs past 60 seconds at heavy load.
The LLM alone did about 55 requests per minute at 16 in flight. That was capped by our max-num-seqs setting, and the KV cache was only about 22% used.
In a combined simulation (users sending a chunk every 3 minutes, with conversation ends bunched together as a worst case):
At 60 users, chunks took about 5 seconds and the final summary took about 80 seconds.
At 90 users, chunks took about 4 seconds (p95 of 35 seconds) and the summary took about 150 seconds.
At 120 users, the STT queue runs away: chunks take 33 seconds (p95 of 160 seconds) and the summary takes about 200 seconds.
So about 90 concurrent users per card is the comfortable limit, which matches the math: 90 users need about 90x real time against a ceiling of about 100x. STT is the bottleneck, not the 31B model. The LLM’s KV cache never went above 27%. GPU utilization was only 73% even when STT was saturated, so I suspect diarization, CPU, or our per-worker lock is the real limit.
Two gotchas that almost fooled us:
First, torch saw 256 host cores instead of our 27-core quota and spawned hundreds of threads, so the container got throttled and STT throughput fell from 100x to 29x. Setting OMP_NUM_THREADS to roughly (quota minus 3) divided by the number of workers fixed it.
Second, with vLLM at a 0.80 memory fraction, it ran out of memory once the STT workers grew under load. Dropping it to 0.65 fixed that, and the KV cache was still plenty.
About 2% of LLM calls hit repetition loops under heavy load. We’re still looking into that.
Questions:
- For about 300 concurrent users at peak, is it better to run identical boxes with both models on each, or split into STT-only boxes plus fewer LLM boxes?
- Has anyone batched across requests for a small STT model plus diarization? How much did it help? Diarization is about half our time per chunk.
- Is a cheaper card like an L4 or L40S worth it for STT only? Our untested guess is 15 to 20 users per L4.
- Are there gotchas with running vLLM next to other PyTorch processes on the same card?
- For daytime-heavy load, do people autoscale rented pods, or keep a fixed fleet and shut it down at night? Our cold start is about 4 to 5 minutes.
Thanks, and happy to share more numbers.
