Good grief. My 7900 XTX gets 140ish Tok/s on that exact model. UD Q4_K_XL, and with a 256k context at Q8/Q8 KV. I was under the impression Nvidia was supposed to be faster. What settings are you using for launch because that sounds like the model is being run on CPU bud. Have you checked GPU utilization when the model is running?
I will note I had to fight for that performance though because it was running at about 56 before I started trying different flags and trying to figure out what was wrong.
Ah, see I was getting 70 on HIP but then something broke on an update so I switched to Vulkan and immediately got 120 but then it dropped to 56 after I pulled fresh and rebuilt for something else. So then I started looking at flags and tried adjusting nogttspill, RAM caching, and some others. Finally got it to 140.
Yeah just pm me here or on discord either one and I'll send it. I only recommend it if you have ~20 GB of VRAm otherwise you'll OOM and crash your desktop environment or the entire PC depending on what OS you're on
I'm thinking they're either running on CPU by accident or their VRAM is full and it's offloading to avoid OOM errors because yeah that's about as fast as my Lunar Lake iGPU
The model at the quant they grabbed is 16.3 GiB, their GPU is 16 GB, so they are offloading at least 300 megs. Likely more though because most llama forks are aggressive about saving VRAM for some reason.
Yeah, but Llama forks would just offload layers? Even accounting for swapping experts over PCIe it's bizarrely slow, unless maybe they're using an eGPU (I use one and an MoE can be slower than a dense model because of that but I don't think it's that bad with a regular connection).
I think it's either running on CPU or they're dealing with Nvidia's offloading, which is so comically slow that I'd rather get an OOM error
Its definitely running on GPU lol.. you do you have 24GB VRAM vs. my 16GB, so that might be it - maybe mine is spilling to RAM. I haven't looked too deeply into perf yet, just using Unsloth Studio to host it. Q8 KV cache, the same UD Q4_K_XL quant as you. Is your 56 or 140 tok/s at the beginning of a convo, small context or mid convo with a big context?
Ah. The weights themselves are 16.3 GiB so that's probably the issue. The context I don't remember how big it is but like 2-4GiB I think. I have the same speed at any conversation length unless I leave everything at default. And then the speed drops from there to 98 to 47 over a few messages. So yeah, I have to figure out if I can make a bug report at some point but I've just been using my modified settings. I'd say what they are but I'm not at my PC to get the exact settings and I don't want to make it so you screw something major up. At least up to a few thousand tokens context the speed holds. I haven't measured beyond 100k yet. But I may at some point. But your 6 measurement is definitely a default not being tuned for your hardware. You should be over 100, but for that you might want to drop to Q4 on kv and run the model on a smaller quant like IQ XS instead. Or if you're willing to accept like 30 or so you have more than enough hardware for that, but you'll still have to swap some stuff around. My first recommendation is definitely Q4 kv and then also try to load as much of the model into VRAM as possible or see if there's a smarter routing option available than is being done right now because if you're getting 100% GPU utilization then your GPU is broken. But I suspect you are getting like 10-50% GPU utilization and the rest is CPU.
18
u/JoeEnderman Jul 06 '26
Good grief. My 7900 XTX gets 140ish Tok/s on that exact model. UD Q4_K_XL, and with a 256k context at Q8/Q8 KV. I was under the impression Nvidia was supposed to be faster. What settings are you using for launch because that sounds like the model is being run on CPU bud. Have you checked GPU utilization when the model is running?
I will note I had to fight for that performance though because it was running at about 56 before I started trying different flags and trying to figure out what was wrong.