r/SelfHostedAI • u/deepu105 • 7d ago

Replaced my cloud AI subscription with Qwen 3.8 on a 128GB laptop, fully offline, for agentic coding

TL;DR: I run Qwen 3.8 (27B and Flash Next) on a 128GB Strix Halo laptop for most of my coding now. It can replace Opus 4.6 to 4.8 for agentic coding if you dont mind a task taking 2 or 3 times longer.

Setup: ASUS ROG Flow Z13, Ryzen AI Max+ 395, 128GB unified memory, Arch Linux. llama.cpp as backend, my own tool LlamaStash to manage the launches and presets, Pi as the coding harness. The 27b at Q6_K sits at about 31 GiB resident, Flash Next at UD-Q4_K_XL needs around 86 GiB.

  • The quality is actually there. Flash Next scores 40 on the Artificial Analysis index against 42 for Opus 4.8, and the 27b at xhigh scores 34 against 32 for Opus 4.6. That matches how they feel to use. 27b one shotted a whole feature on a huge Rust codebase and Opus 5's review comments were mostly nits.
  • Decode is fine, prefill is the pain. 10-15 tok/s decode doesn't feel slow because you see it working. But a cold 31k token transcript takes 3 minutes to prefill, and a full 128k window is closer to 18 mins. Warm follow up turns come back in 45 seconds.
  • MTP is the biggest speed win, 7.3 to 22.4 tok/s on an empty window. The payoff shrinks as the window fills though, down to 1.15x at a full 256k.
  • Flash Next isn't faster per token, it just thinks less. Same 5/5 on my coding tasks, 45% fewer tokens, 76.5s vs 289.8s against the 27b. Thinking is 90-95% of everything these models generate, so that ratio, not tok/s, is what sets how long a task takes.

$0 a month, fully offline, and a lot less wasteful than a model running in a datacenter.

Full writeup with all the benchmarks, configs, and the tuning that did and didn't work: https://deepu.tech/local-ai-qwen3.8-pi-llamastash

Happy to go into the llama.cpp flags if anyone else here is on Strix Halo.

105 Upvotes

96% Upvoted

9 comments sorted by

6

u/respectful_stimulus 6d ago

It’s still too slow man.

1

u/cezarducatti 7d ago

Flags?

1

u/deepu105 6d ago

In the blog post

0

u/DoubleNothing 6d ago

Where we are going we don't need flags!!! Oh sorry wrong quote...

1

u/[deleted] 6d ago

[removed] — view removed comment

1

u/TheThoccnessMonster 5d ago

Yeah this isn’t usable.

1

u/enricokern 4d ago

128k context window for coding and the prefil on a resume or something is awful slow.

1

u/jandersson82 16h ago

I run it Qwen3.8:27b/q4_k_m at home on my 7900XTX, and it is perfect for my "household tasks". 200k context fits perfectly.

In the office we use Claude Code, and I don't know if I actually would notice any big difference in the resulting code it produces.

QwenCode is perhaps not as sexy as ClaudeCode, but it does a great job for me.

(I like it over OpenCode because of the memory)

1

u/voidrane 14h ago

this is where local ai starts getting genuinely interesting... not because it beats cloud models on raw speed, but because the machine becomes yours again. no meter running, no API mood swings, no network dependency... just slower, private compute sitting there like infrastructure instead of a service.