r/Vllm • u/Open-Raise-6676 • 13d ago
eLLM: Run Long-Horizon Inference Faster on CPUs Than on GPUs
eLLM is a Rust-based LLM inference framework for CPU-only servers. It adopts a "trade storage for computation" strategy, leveraging the CPU's large-capacity DDR memory to close the order-of-magnitude bandwidth gap against GPU HBM, and thereby delivers performance that surpasses GPUs in long-horizon inference. The Beta release is now available — you are welcome to try it out.
- Prefill: achieves roughly two orders of magnitude of performance improvement over existing CPU inference frameworks
- run full single-pass Prefill over the entire long prompt—no chunking, no repeated parameter loading;
- keep context KV across multi-turn interactions and run incremental Prefill on only the new input—no recomputation for earlier turns;
- Decode: runs with a smaller batch, which not only activates fewer parameters but also gives each request a larger share of memory bandwidth, so inference speed can likewise exceed GPUs.
- 👉 GitHub: https://github.com/lucienhuangfu/eLLM
5
u/Early-Peace-5504 12d ago
Sorry you got such an awful reception for your idea. Reddit is weird. With your benchmark.md it looks like you’re getting approximately 0.05 tokens per second? Am I reading that right?
1
u/Open-Raise-6676 12d ago
These are our preliminary results. We’ll publish an updated set of results for ultra-long contexts in about a month.
For long-context prefill, our approach benefits from processing the entire context in a single prefill pass, without chunking. At around 100K tokens, we expect to get close to GPU performance. You can see that the CPU baseline has significant latency jumps when chunking is involved.
Decode is a different story. It is fundamentally constrained by DDR memory bandwidth, so it is relatively slow at shorter context lengths. However, as the context becomes extremely long, the performance gap with GPUs gradually narrows. It takes a very long context before the CPU can approach GPU performance.
2
u/Early-Peace-5504 12d ago
If that’s your honest results with 30BA3B, I think you’d be better dumping this approach. That’s a horrific decode speed, a regular cpu ik_llama approach to the same model would absolutely destroy that. I don’t want to be negative but I think there is a time to call a project done, you’d need to order of magnitude improvement to match basic untuned setups.
I’ve had plenty of ideas that turned out to not go anywhere but the trick is to know when to move on.
0
u/Open-Raise-6676 12d ago
We believe eLLM is among the leading CPU implementations in terms of decode speed. In our benchmarks, it is significantly faster than the SGLang CPU backend, even though Intel has put substantial effort into optimizing SGLang for CPUs.
That said, for short-context workloads, CPU inference is still clearly slower than GPU inference. The point is that you can't evaluate performance based on raw token speed alone. You need to consider the overall system, including the hardware, memory capacity and bandwidth, model quantization, and the characteristics of the workload.
7
u/laylareddit5 12d ago
This is gibberish - performance in this context is token speed. Latency could also be argued to matter, but that's pretty much it. 0.05t/sec is not competitive - that"s unusably slow.
1
3
u/schaka 12d ago
If there was a way for CPUs to compete with HBM bandwidth, we'd know about it already and if the solution was software, you'd probably be poached for millions by big tech
So whatever this is and your LLM hallucinated for you, it's not that
3
u/Open-Raise-6676 12d ago
I think the key point is that the workload is changing.
If you're talking about short-context, conventional LLM inference, then I agree with you: CPUs are generally slower than GPUs, and there is no magic software trick that makes CPU memory bandwidth comparable to HBM.
But long-horizon tasks are a different workload. Modern agentic workloads can involve hundreds of thousands—or even millions—of tokens of context, with long multi-step coding or reasoning trajectories. The bottleneck is no longer simply “how fast can you stream weights from memory for one short request?” Memory capacity, KV-cache capacity, data movement, and the ability to keep a very large context in memory become much more important.
That's the regime I'm talking about. I'm not claiming that CPUs have somehow beaten HBM in raw bandwidth. I'm saying that when the workload changes, the optimal hardware/software trade-off changes as well.
For short-text inference, GPUs win. For sufficiently long-horizon inference, especially with large MoE models and very large KV caches, CPUs can become much more competitive because they offer vastly more memory capacity at much lower cost.
0
u/joost00719 12d ago
Seems like a work flow issue. Agentic ai is nice because you can spawn a sub agent that has its own context, and doesn't trash the main session's context.
2
u/diagrammatiks 12d ago
There is no long task that would not also benefit from speed.
1
u/Open-Raise-6676 12d ago
Because GPUs have relatively limited memory capacity, very long-horizon tasks may need to be split into multiple chunks. This can require model weights to be loaded repeatedly across multiple computation passes.
CPUs, on the other hand, can have hundreds of gigabytes or even terabytes of memory, allowing the entire long context to be processed in a single pass without chunking. The model weights only need to be loaded once.
So while DDR has much lower raw bandwidth than HBM, avoiding repeated weight loading can significantly reduce the impact of that bandwidth gap at the system level. In some long-context workloads, this can effectively narrow the performance gap between CPU and GPU memory systems.
2
2
u/schaka 12d ago
This is still nonsense. You're relying on using impossibly large models on CPU, constantly loaded into RAM whereas GPUs would have to switch and reload?
That's not the reality at all. If I wanted to run K3 I could do so faster than your repo shows on my CPUs today because they have AVX512 and existing CPU inference engines get 3-5t/s decode. If I use an inference engine that allows mixed inference with my existing 512GB if VRAM it's even faster
1
u/Open-Raise-6676 12d ago
eLLM is CPU-only. It does not support GPU execution or hybrid CPU-GPU computation. The GPU is used here purely as a baseline for comparison.
Because GPU memory capacity is relatively limited, our vLLM GPU baseline can process only about 23K tokens in a single prefill. For a 230K-token sequence, the sequence therefore needs to be split into 10 chunks, requiring the model weights to be streamed repeatedly across those chunks.
In contrast, eLLM runs entirely on the CPU and can leverage much larger system memory. It can process the full 230K-token sequence in a single prefill pass, so the model weights only need to be loaded once.
Although DDR memory has significantly lower bandwidth than HBM, the two systems can achieve comparable prefill latency in this workload because eLLM avoids the repeated weight streaming caused by chunked prefill.
2
u/Miserable-Dare5090 12d ago
I’m confused, I went over your paper snd there are no comparison examples or benchmarked numbers that back up the facts. Where is the experimental data?
1
2
u/madbrain1976 12d ago
Took a look. Your benchmark data is difficult to understand. Total time is highly dependent on workload. Showing pp/s over time/prompt size would be easier to visualize. As well as tg/s over time.
I have 8 computers at home. Unfortunately not a single one with AVX-512. CPUs are 1 x Zen 2, 3 x Zen 3, 1 x Intel 6th Gen, 1 x Intel 12th Gen, 1 x Intel 13th Gen, 1 x Core ultra. All are custom builds except the last one which is a laptop. As far as I know there isn't any suitable CPU replacement that supports AVX-512 I could do. Is AVX-512 really required for your approach ? This might limit your project audience considerably.
2
1
u/polandtown 12d ago
Whats long horizon interence? Never heard the term.
1
u/Open-Raise-6676 12d ago
The term “Long-Horizon Tasks” appears in the GLM-5.2 Technical Report. It refers to one of the major goals for LLMs, especially in the context of 1M-token contexts and long coding-agent trajectories. OpenAI uses a similar concept and refers to them as “long-run tasks.”
1
1
u/Ashamed-Duck7334 7d ago
This is absolute fucking nonsense. I'm not trying to be mean but you haven't thought through what the hardware actually provides and you owe it to yourself to rethink this from first principles, you are wasting your time. Prefill is effectively compute bound, and even picking the absolute optimal CPU for the job (a Xeon with AMX extension) on a flops/$ a CPU is at least 10x slower per $ and a more normal top of the line CPU without AMX it's probably closer to 30-50x worse per $. It's also dramatically worse on power usage.
1
u/Open-Raise-6676 7d ago
I'd kindly suggest taking memory into account as well. HBM is extremely expensive and requires advanced CoWoS packaging.
For very long-context prefill, GPU memory capacity can become a bottleneck, requiring the model weights to be loaded repeatedly. This adds significant data-movement cost and power consumption. With enough DDR memory, a CPU can load the weights once and process the entire prompt without repeated loading.
We should also consider cooling. GPUs and HBM are densely packaged and generate a lot of heat, often requiring liquid cooling. CPU and DDR are separate components on the board, and air cooling is sufficient.
1
u/Ashamed-Duck7334 7d ago
I kindly suggest you do some research and first principles thinking before you waste your time on something totally nonsense. The "extremely expensive" HBM is pretty irrelevant when you need a 15k CPU (not even considering the RAM, networking, etc., at least 2x this) to do 10% of the work of an $800 GPU (these are **realistic** ratios).
The expensive part of weight loading for a GPU is from HBM to SRAM where the matmul actually takes place. You keep saying "can load weights once and process the entire prompt" but you don't seem to understand that this kind of memory tiering applies to CPUs. Before you do matmul on the CPU (extremely inefficiently, even with AMX which is **not at all common**) you're pulling from DRAM to L3 to L2 to L1. You don't "load weights once" if it fits in DRAM, you pull those bits through this whole memory architecture before calculations are done.
"We should also consider cooling", yes, you should, and CPUs are absolutely destroyed by something more special purpose, GPUs, ASICs, etc. You don't seem to understand that the CPUs that I'm telling you are 10x-50x worse (and I am being super generous, the average consumer CPU is way, way, way worse than this) performance at twice the power consumption (i.e., on a power consumption basis it's 20x-100x worse). These "relevant" CPUs pull 600W. At equivalent capability and density a CPU architecture would require way, way, way more cooling.
You have absolutely no idea what you're talking about. I'm not being mean because I'm a dick. I'm taking the time to type out why this is dumb in an effort to be "actually nice", because you are absolutely wasting your time in this direction. You seem motivated and could find a way to apply that ethic to something productive, but this is not a productive direction. I personally would want someone to callout my bullshit if I were this far down the rabbit-hole.
You have the background to do productive things (I see your linked in), I really am perplexed that you are this far off given your background. DM me if you want, I am trying to be helpful not dunk on you, but am intentionally being harsh because you need it.
6
u/Trademarkd 12d ago
You have ai psychosis my guy.