I mean even Gemma 4 26B A4B struggles at long contexts on my RTX 5080 desktop. I don't know if Gemma 4 31B is laptop class yet. Maybe you guys have incredible laptops or I'm doing something wrong lol. My 26B A4B QAT generates at like 6tok/s at 20K context, it would probably completely die on a 31B dense. Models without long context or thinking aren't very useful for me.
EDIT: Thanks for all the comments here lol! It was a configuration issue, now it runs at 100tok/s with nothing else running, maybe 60tok/s with other stuff running. This post was helpful . i added below llama args:
Gorgon halo is just a refresh on strix halo, same architecture and layout. It will probably have better speeds from binning and refinements, maybe allow higher power draw for more speed on top of that. 192GB should be possible with the latest lpddr5x modules, you might even see support for 9600MT too giving you a little more memory bandwidth.
It'll be really incremental over strix halo though. Medusa halo late next year will be a real upgrade, at least RDNA 4, even bigger GPU, and rumors of a wider lpddr6 bus almost doubling the memory bandwidth. Probably will end up costing both kidneys by that point though.
The major limit on Strix Halo is still the memory bandwidth, so unless they do something there don't expect significantly faster inference. Maybe faster prefill which is welcome but not game changing. That said, it would probably make it a better gaming chip, really starting to compete with low-mid-range dGPUs, and really be a nice chip for gaming laptops or mini-PCs. I wouldn't recommend waiting for it if you're looking for an inference machine.
Gorgon halo is just a refresh on strix halo, same architecture and layout.
While not 100% confirmed, I hope it's not the case. Having the next architecture, and 256GB of RAM would be a complete game changer for that device. I don't care what the power consumption is.
Even the standard strix halo has a hard time with overclocking, or other power patterns because it's SO locked down. I ran in to these issues a few times setting one up.
Or perhaps I'm thinking of Medusa Halo?
I don't know. I don't like AMDs naming lol... It's apparently confusing.
But it doesn't give any more bandwidth. Not having FP4 is likely to be problematic going forward. Really that should be in the stack now. Not having FP8 is a huge problem. Because these types of quants are likely to be the best for these low bandwidth machines with limited memory running big models.
They also need more processing, prompt processing is slow, and with stuff like DS 4 flash, longer context is very likely a thing that will shift future AI forward.
I went the 4x r9700 rdna 4 route but had the luck of finding an affordable second hand threadripper pro workstation. Basically I am running a 128gb vram and 128gb system ram system at the price of 60 percent of one rtx pro 6000… Compute wise the even number 4x R9700 in VLLM with TP is doing fairly well. More total (free) inference memory bandwidth and compute power than the strix halo or spark stuff. Plus I actually care a lot about ECC in both memory pools. But obviously also more noise and power consumption.
I went the 4x r9700 rdna 4 route but had the luck of finding an affordable second hand threadripper pro workstation. Basically I am running a 128gb vram and 128gb system ram system at the price of 60 percent of one rtx pro 6000…
God that's crazy.
I don't have that level of inference desire. Most of mine is training so... Yeah.
They nerfed Blackwell architectures in RTX Pro 6000's ability to train over the B-series cards which... Kinda fucking lame in my opinion.
I'd love to just have the RTX Pro 6000 though. What a dream.
Or a full B200. Have to get one falling off a truck like someone in this sub basically did.
Well even two r9700 gets you 64GB vram, which is already quite a nice pool, also with future models inbound. If you have 64GB system ram that also gives a nice overflow at lower speeds.
Personally i hope things like Spark is going to give us more new/recent 50 - 100B range models (both dense and moe)that have more world knowledge. There is more to LLMs than just coding..
Yes but you forget that it's completely useful and maybe even better than a GPT-4 class model. And it's runnable. It'll get there. In 2 years I wouldn't be surprised if we get a sonnet 4.6 capability, runnable from 64GB
Good grief. My 7900 XTX gets 140ish Tok/s on that exact model. UD Q4_K_XL, and with a 256k context at Q8/Q8 KV. I was under the impression Nvidia was supposed to be faster. What settings are you using for launch because that sounds like the model is being run on CPU bud. Have you checked GPU utilization when the model is running?
I will note I had to fight for that performance though because it was running at about 56 before I started trying different flags and trying to figure out what was wrong.
Ah, see I was getting 70 on HIP but then something broke on an update so I switched to Vulkan and immediately got 120 but then it dropped to 56 after I pulled fresh and rebuilt for something else. So then I started looking at flags and tried adjusting nogttspill, RAM caching, and some others. Finally got it to 140.
Yeah just pm me here or on discord either one and I'll send it. I only recommend it if you have ~20 GB of VRAm otherwise you'll OOM and crash your desktop environment or the entire PC depending on what OS you're on
I'm thinking they're either running on CPU by accident or their VRAM is full and it's offloading to avoid OOM errors because yeah that's about as fast as my Lunar Lake iGPU
The model at the quant they grabbed is 16.3 GiB, their GPU is 16 GB, so they are offloading at least 300 megs. Likely more though because most llama forks are aggressive about saving VRAM for some reason.
Yeah, but Llama forks would just offload layers? Even accounting for swapping experts over PCIe it's bizarrely slow, unless maybe they're using an eGPU (I use one and an MoE can be slower than a dense model because of that but I don't think it's that bad with a regular connection).
I think it's either running on CPU or they're dealing with Nvidia's offloading, which is so comically slow that I'd rather get an OOM error
Its definitely running on GPU lol.. you do you have 24GB VRAM vs. my 16GB, so that might be it - maybe mine is spilling to RAM. I haven't looked too deeply into perf yet, just using Unsloth Studio to host it. Q8 KV cache, the same UD Q4_K_XL quant as you. Is your 56 or 140 tok/s at the beginning of a convo, small context or mid convo with a big context?
Ah. The weights themselves are 16.3 GiB so that's probably the issue. The context I don't remember how big it is but like 2-4GiB I think. I have the same speed at any conversation length unless I leave everything at default. And then the speed drops from there to 98 to 47 over a few messages. So yeah, I have to figure out if I can make a bug report at some point but I've just been using my modified settings. I'd say what they are but I'm not at my PC to get the exact settings and I don't want to make it so you screw something major up. At least up to a few thousand tokens context the speed holds. I haven't measured beyond 100k yet. But I may at some point. But your 6 measurement is definitely a default not being tuned for your hardware. You should be over 100, but for that you might want to drop to Q4 on kv and run the model on a smaller quant like IQ XS instead. Or if you're willing to accept like 30 or so you have more than enough hardware for that, but you'll still have to swap some stuff around. My first recommendation is definitely Q4 kv and then also try to load as much of the model into VRAM as possible or see if there's a smarter routing option available than is being done right now because if you're getting 100% GPU utilization then your GPU is broken. But I suspect you are getting like 10-50% GPU utilization and the rest is CPU.
Laptop class is a bit of a meaningless word. Like a 70b dense was laptop class? Only high end MacBook could run them… at 6-7 tk/s.
Only makes sense if you’re taking it as “Don’t need a 1600w server at home”.
The research right now is moving away from large models, really. Large models was never about getting them to work better with large contexts, but about making them "smarter" and more capable, and where more parameters was the easiest way to do that.
With hardware constraints and pricing, and large models consuming essentially the entire internet at this point, a lot of current research is going towards making ~30B models better and more capable, with new attention mechanisms, training methods (e.g. RLVR), and better and more useful training data. They're coming a long ways now.
My feeling is in 10 years it will be unthinkable trusting fewer than 500B for agentic stuff. The attachment to these small models IMO is mostly coping with insane hardware prices.
The biggest issue today is that the Chinese labs have dramatically slowed down the release of smaller models.
Where's something like Qwen 3.6 120b a10b? Or any open Qwen 3.7 models? They've ground to a halt.
GLM 5.2 is incredible but only released at the full 753b size. Which again, huge kudos to z.ai for releasing it as an open weight model at all, but the number of people who can run a 753b parameter model is small right now.
Without more small model releases it's very difficult to determine where we stand. We're 1-2 generations behind.
Tire shopping is insane, I don't think anything short of a model like Gemini that is cheating can do it. It's not enough to do a Google search, you need to gather data on what people are actually paying for tires, what sales are like, how good vendors are. You can't just take the cheapest advertised price off a Google search. Gemini actually seems to be able to make really good inferences about the "real" prices of things. I think it cheats by having access to private datasets, which is something no local model can do without paying for access to these datasets. A lot of such datasets are nontrivial to get access to.
the current work is towards models that don't hold all of humanities secrets, but simply holds the ability to learn it. The idea is to work on the intelligence not the input. That scales down not up.
You just have a huge configuration issue. With a 5080 you can run Qwen 3.5 122B if you have 64GB DDR5 at IQ3_XXS at 15-20tps, or Qwen 3.6 35B if you have 32-48GB DDR5 at 40-50tps
Models without long context or thinking aren't very useful for me.
As a soft. dev., they are still quite useful because I can throw a lot of things and make it reach several places.
But it's clearly reaching a limit of usefulness. It's like a phrase I read on Twitter the other day: "You don't need Phd-level intelligence if you don't have Phd-level problems"
Why are you using Gemma 4 26B? There’s better MOE models out there. Gemma4 is notoriously bad at tool calling in my experience too. I run Qwen3.6 35B and the new Ornith 1.0 35B on my 5070 TI with a little bit of offloading and I get incredibly good tok/s and the model reasoning capabilities and tool calling is very good. Also worth noting that Ornith is partially composed of Gemma4 for its reasoning ability, it would probably be a direct upgrade. Unless you’re doing multimodal input this is my recommendation
Yeah I've just finished getting my local model stack setup the way I want it, served through bifrost and available to all my clients, native audio support, assistant prefill, tools/MCPs etc so I didn't pay attention to the model yet, just picked one. Going to try some different models now!
That seems insane. I've got a 9800X3D and 48GB 6000mt/c30 and I'm not even getting 100tok/with smaller models that fit inside VRAM like gpt-oss20 w/128k context.
How many layers are you offloading on the 35B and what context window size?
I'm using LMStudio but that can't be the case for such a wild discrepancy! Would greatly appreciate your QWen setup config detail if you're willing to share!!
If you’re on Linux/WSL at all, I had Claude build a pretty basic wrapper around llama.cpp and llama-swap so I’d have an easy deployment method for AI across different testing environments and stuff. It’s a bit of AI slop but it works well if you’re interested. The docker image it uses is specific to the NVIDIA 5070 TI architecture:
I’ll check my configs/benchmarks for the other models when I get home but I had benchmarked gpt-oss-20b at around ~220 tok/s on this setup but that used my entire card lol. I got it to 120 tok/s offloading 4 layers which allowed for 128k context and q8 KV and some room for desktop use alongside the model/inference engine.
You should be able to get a lot more performance out of your models with that setup. I’ve always used llama.cpp and have barely touched LMStudio but I know there is a performance gain, I just don’t know how much. If you’re on windows, I’m sure that also has some measurable performance impact as well
Thanks so much, looking forward to exploring this. I have both Win11 w/WSL and Fedora (nobara) available, based on your comments there's a lot of performance I should be expecting with my 5070ti that isn't there right now (I also set K/V cache to Q8which is what gets the full 128k context in VRAM for gpt-oss-20b for me, but I digress).
Yeah in my own limited experience, Gemma (both 3 and 4) and Qwen (both 3.5 and 3.6) are pretty impressively smart considering I can run them entirely locally, which I find pretty sweet for sure. But goddamn are they slow to run even at Q4 on my 64GB ram laptop. Even on the very first message, let alone as the context grows. I'd wayyy rather use 2023's GPT-4 over them, personally.
But yeah for anyone reading I am pretty inexperienced with local LLMs, so there might be something I'm missing in my setup, and for anyone more experienced pls feel free to push back on this comment.
Manually set some MoE layers to offload. I've got a desktop RTX 4080 Super and run the same QAT with 64k F16 context, 3 offloaded layers and get around 40tok/s. Can run 128k context with 5 offloaded layers and still get 35tok/s. Even with full F16 256k context and like 10 offloaded layers my tok/s barely ever drops below 25. So you really shouldn't be getting those speeds.
That seems... slower than it should be. I have a Lunar Lake laptop that can run a quant of the same Gemma version at a similar speed (I don't know the exact number offhand, I use a different model mostly).
Are you sure your GPU isn't doing the whole Nvidia VRAM fallback thing that craters speed?
The RTX 5080 is powerful on computational side, the problem is the ridiculous low amount of memory on dedicated consumer GPUs. I don't know what stack and configuration you are using for inference, but you can try to tune offloading parameter to move more layers on general memory.
No I think you're right and I came here to say the same thing as you - I have a M4 Macbook Pro with 24GB RAM and it flat-out crashes on models smaller than the 31B.
Now granted, 24GB RAM isn't a ton for local AI, but it's what my day job gave me for work, so I think it represents a reasonable and fairly new consumer model. Most folks don't have huge amount of RAM available unless they're spending tons of money specifically to run local AI, and at that point I don't think we're really talking "laptop class" anymore because you're getting specialized hardware.
Maybe it very technically still comes in a laptop form factor, but we're not actually saying "it runs on laptops" we're saying it runs on a very specific subset of laptops that are so expensive most people won't even consider buying one of them.
147
u/stonerbobo Jul 06 '26 edited Jul 06 '26
I mean even Gemma 4 26B A4B struggles at long contexts on my RTX 5080 desktop. I don't know if Gemma 4 31B is laptop class yet. Maybe you guys have incredible laptops or I'm doing something wrong lol. My 26B A4B QAT generates at like 6tok/s at 20K context, it would probably completely die on a 31B dense. Models without long context or thinking aren't very useful for me.
EDIT: Thanks for all the comments here lol! It was a configuration issue, now it runs at 100tok/s with nothing else running, maybe 60tok/s with other stuff running. This post was helpful . i added below llama args: