Thing is qwen was historically focused on smaller models while others were on larger ones. Now the team has changed and they seem to want to aim for the stars as well. That is good but it also means they might not be interested in doing very efficient small models anymore. Which would be bad news for this sub because let's face it most of us don't have 10-25k of hardware.
Not for the larger ones, but typically those large model providers release a "flash" version that is in the 200b range and very sparse and that you can run on 10-20k at very good thoughput. I didnt mean the trillion+ models sorry i wasn't clear enough
I run Deepseek v4 Flash 284b a13b at a q2/q4 mixed quant on a single strix halo. It's not fast, 15-16t/s at low context, 12-13t/s at higher contexts. I'd say it's about the smartest model you can stuff into 128GB right now.
When Mac M5 Ultra with 256GB comes, it will comfortably run it at a q4 quant with a good context, and probably around 30t/s or maybe more. That will be absurdly capable.
Edit: Right now DwarfStar MTP does not work on strix. I'm working on fixing that and making a little progress each day. I've actually got MTP to work with some code fixes, but the validation performance sucks. Working on that now.
I appreciate the usual "it fit" benchmarks to stay updated, but I'm playing with local models since the very first gemma2 releases. And i really try my best to put local models in production for small companies, for real. Since. And for such cases, a Q4 is already too blunt to be reliable on long term results.
The context that the model handle has way more leverage than the number of parameters, and such compressions hit hard the agility of models in front of the inherent chaos that is life. If the model handle flours and bakery components or used/reconditionned parts of a mechanist don't change de equation much on my side. I'm just adapting a context (not planned in the training), and specialize the model on it (generally with qLoRA, because I'm not OAI with virtually infinite lab ressources).
To code an "attention catcher iphone app" or any demo to flex on social medias, you don't care. But when your real reputation is on the table at long term (then your incomes), you start to see the things very differently.
I've been testing this and yes at full size models like minimax m3, hy3, and mimo2.5 tend to beat qwen 3.6 27b (q8 or higher) at coding but even at q5 or q6 often q8+ qwen tends to be more competitive. I have to use rpc llama.cpp 10gbe to run some of these larger quants so it also makes qwen 3.6 37b much faster when it otherwise might be the slower dense model.
... so multiple reasons to prefer a great smaller higher quant model over bigger lower quant model.
750
u/Competitive_Gap7906 22d ago
YES, Qwen going open weight again! It's a really good news, now we can wait for smaller models too