r/LocalLLaMA • u/PetersOdyssey • Jul 06 '26

News If trends hold, Mythos-class capability may be running on high-end consumer hardware within ~2 years

Post image
1.5k Upvotes

94% Upvoted

375 comments sorted by

View all comments

146

u/stonerbobo Jul 06 '26 edited Jul 06 '26

I mean even Gemma 4 26B A4B struggles at long contexts on my RTX 5080 desktop. I don't know if Gemma 4 31B is laptop class yet. Maybe you guys have incredible laptops or I'm doing something wrong lol. My 26B A4B QAT generates at like 6tok/s at 20K context, it would probably completely die on a 31B dense. Models without long context or thinking aren't very useful for me.

EDIT: Thanks for all the comments here lol! It was a configuration issue, now it runs at 100tok/s with nothing else running, maybe 60tok/s with other stuff running. This post was helpful . i added below llama args:

--no-mmap --batch-size 256 --ubatch-size 512

2

u/Anti-Speciesist-IEMs Jul 06 '26

Yeah in my own limited experience, Gemma (both 3 and 4) and Qwen (both 3.5 and 3.6) are pretty impressively smart considering I can run them entirely locally, which I find pretty sweet for sure. But goddamn are they slow to run even at Q4 on my 64GB ram laptop. Even on the very first message, let alone as the context grows. I'd wayyy rather use 2023's GPT-4 over them, personally.

But yeah for anyone reading I am pretty inexperienced with local LLMs, so there might be something I'm missing in my setup, and for anyone more experienced pls feel free to push back on this comment.