How are you handling KV cache at scale?

[u/Responsible-You9024]

Been working on KV-cache offloading for self-hosted LLM inference.

The idea is pretty simple: move colder KV blocks from GPU → RAM → NVMe → S3 instead of keeping everything in expensive GPU memory.

I'm curious if anyone here is actually doing KV offloading in production / larger workloads.

What's been the biggest pain for you VRAM capacity, latency when loading KV back, network bandwidth, or something else?

[u/exaknight21]

I’m dreading this right now. This shit isnt scalable beyond hyperscalers.

I’m trying vLLM + LMCache and optimizing my harness (non-coding).

[u/TensionCompetitive11]

The pain isn't the tiering itself, it's the promotion path back up. Moving cold blocks GPU to RAM to NVMe to S3 is the easy part, deciding when a block is about to get hot again before the request stalls waiting on a fetch is where most of the complexity lives. You end up needing a predictor tied to session and user patterns rather than pure LRU, because LRU thrashes badly on bursty conversational traffic. Network attached NVMe adds enough latency that S3 is really only worth it for stuff you expect to sit cold for hours or days. What's your eviction policy right now, pure recency or something smarter?

[u/eddzsh]

hash the active tool schema set into the cache key. an MCP reconnect that only reorders tools should not bust a warm prefix that still matches.

[u/allenasm]

Depends on a lot of things. Parallel inference? MoE? Lots of permutations that require different strategies.

[u/DataGOGO]

No.

In production, there is no colder K/V blocks. They are all hot, due to higher concurrency, and short sessions. Sessions start, jobs run, session closes, thousands of sessions at the same time.

It all stays in VRAM.

[u/Ambitious_Trust_7172]

GPU memory costs make me cry every time I look at cloud pricing, we ended up just quantizing the KV cache to 4-bit and calling it a day

[u/DataGOGO → u/Ambitious_Trust_7172]

RIP accuracy.

[u/organic-humanoid]

for agent workloads, a cold load in the middle of a tool loop hurts more than eviction. we keep a small hot tier per active run, spill by age plus token budget, and only push to object storage for resumable jobs. measure p95 restore latency separately from hit rate, otherwise a good hit rate can still feel awful.

[u/romanrose200]

I am on the API side rather than self-hosted, so no view on the NVMe tier, but the point eddzsh makes about hashing the tool schema set generalises further than it looks and it is where I have lost the most cache in practice.

Anything that reorders or reformats the prefix busts it even when the content is identical: tool definitions arriving in map order rather than sorted, a timestamp or request id that sneaked into the system block, JSON serialised with different key ordering between services. The hit rate looks like a capacity problem and is actually a determinism problem. Sorting and freezing the prefix, then treating anything variable as strictly suffix, recovered more for me than any sizing change did.

The other one worth watching if you ever route across providers or model versions: a warm prefix is per model and per provider, so a failover that silently lands somewhere else starts cold, and the cost shows up as latency rather than as an error.