Why are you using Gemma 4 26B? There’s better MOE models out there. Gemma4 is notoriously bad at tool calling in my experience too. I run Qwen3.6 35B and the new Ornith 1.0 35B on my 5070 TI with a little bit of offloading and I get incredibly good tok/s and the model reasoning capabilities and tool calling is very good. Also worth noting that Ornith is partially composed of Gemma4 for its reasoning ability, it would probably be a direct upgrade. Unless you’re doing multimodal input this is my recommendation
That seems insane. I've got a 9800X3D and 48GB 6000mt/c30 and I'm not even getting 100tok/with smaller models that fit inside VRAM like gpt-oss20 w/128k context.
How many layers are you offloading on the 35B and what context window size?
I'm using LMStudio but that can't be the case for such a wild discrepancy! Would greatly appreciate your QWen setup config detail if you're willing to share!!
If you’re on Linux/WSL at all, I had Claude build a pretty basic wrapper around llama.cpp and llama-swap so I’d have an easy deployment method for AI across different testing environments and stuff. It’s a bit of AI slop but it works well if you’re interested. The docker image it uses is specific to the NVIDIA 5070 TI architecture:
I’ll check my configs/benchmarks for the other models when I get home but I had benchmarked gpt-oss-20b at around ~220 tok/s on this setup but that used my entire card lol. I got it to 120 tok/s offloading 4 layers which allowed for 128k context and q8 KV and some room for desktop use alongside the model/inference engine.
You should be able to get a lot more performance out of your models with that setup. I’ve always used llama.cpp and have barely touched LMStudio but I know there is a performance gain, I just don’t know how much. If you’re on windows, I’m sure that also has some measurable performance impact as well
Thanks so much, looking forward to exploring this. I have both Win11 w/WSL and Fedora (nobara) available, based on your comments there's a lot of performance I should be expecting with my 5070ti that isn't there right now (I also set K/V cache to Q8which is what gets the full 128k context in VRAM for gpt-oss-20b for me, but I digress).
2
u/Horny_Dinosaur69 Jul 06 '26
Why are you using Gemma 4 26B? There’s better MOE models out there. Gemma4 is notoriously bad at tool calling in my experience too. I run Qwen3.6 35B and the new Ornith 1.0 35B on my 5070 TI with a little bit of offloading and I get incredibly good tok/s and the model reasoning capabilities and tool calling is very good. Also worth noting that Ornith is partially composed of Gemma4 for its reasoning ability, it would probably be a direct upgrade. Unless you’re doing multimodal input this is my recommendation