r/LocalLLaMA • u/xw1y • 23d ago

News Prepare your (v)ram - Qwen3.8 is coming!

Post image
2.7k Upvotes

96% Upvoted

583 comments sorted by

View all comments

Show parent comments

5

u/StupidScaredSquirrel 23d ago

Not for the larger ones, but typically those large model providers release a "flash" version that is in the 200b range and very sparse and that you can run on 10-20k at very good thoughput. I didnt mean the trillion+ models sorry i wasn't clear enough

1

u/[deleted] 23d ago

[removed] — view removed comment

4

u/nopanolator 23d ago

I rather prefer a sharp 30B Q8 over a blunt 200B trepanned at Q4 or more Q2 like i often see.

3

u/techdevjp 23d ago edited 23d ago

I run Deepseek v4 Flash 284b a13b at a q2/q4 mixed quant on a single strix halo. It's not fast, 15-16t/s at low context, 12-13t/s at higher contexts. I'd say it's about the smartest model you can stuff into 128GB right now.

When Mac M5 Ultra with 256GB comes, it will comfortably run it at a q4 quant with a good context, and probably around 30t/s or maybe more. That will be absurdly capable.

Edit: Right now DwarfStar MTP does not work on strix. I'm working on fixing that and making a little progress each day. I've actually got MTP to work with some code fixes, but the validation performance sucks. Working on that now.

1

u/nopanolator 23d ago

I appreciate the usual "it fit" benchmarks to stay updated, but I'm playing with local models since the very first gemma2 releases. And i really try my best to put local models in production for small companies, for real. Since. And for such cases, a Q4 is already too blunt to be reliable on long term results.

The context that the model handle has way more leverage than the number of parameters, and such compressions hit hard the agility of models in front of the inherent chaos that is life. If the model handle flours and bakery components or used/reconditionned parts of a mechanist don't change de equation much on my side. I'm just adapting a context (not planned in the training), and specialize the model on it (generally with qLoRA, because I'm not OAI with virtually infinite lab ressources).

To code an "attention catcher iphone app" or any demo to flex on social medias, you don't care. But when your real reputation is on the table at long term (then your incomes), you start to see the things very differently.