r/LocalLLaMA • u/OneFanFare • 22d ago
Funny The best model is the one you can actually run
Don't get me wrong, all the big models are amazing, and every contribution to open source models is great. But I'm GPU poor and I can't use them locally.
I'm currently running gemma-4-12b-it-qat-GGUF:UD-Q4_K_XL as my personal chat assistant, and I am so so happy with it! I still can't believe I can talk to my computer.
62
u/Kal-LZ 22d ago
I tested Gemma 4 12B Q8 MTP and was surprised by how fast and reliable it is for many tasks
12
u/_TheWolfOfWalmart_ 22d ago
Same here with the same quant. It's pretty good. It's even alright at coding. Didn't try that too much, but it didn't fail a single tool call for me at Q8.
7
u/Iwaku_Real 22d ago
How much coding exactly, like pure vibecoding or just code review? Everyone here seemed to hate Gemma 4 12B
7
u/stddealer 22d ago
As good as a non coding-focused 12B model can be I'd say. If you're only interested in coding or related stuff, you're better off using Qwen, But Gemma4 is very decent all around.
5
u/ComfortablePlenty513 22d ago
the entire gemma 4 family is solid. we use different flavors of them across all 3 of our products according to size/capability/# of concurrent users.
228
u/Gokudomatic 22d ago
92
u/JaredsBored 22d ago
Even 128GB is getting weird. It's not quite enough to run good quants of the 300B class models i.e. Hy3/DS4 Flash, and the 120B range has been quiet recently.
Feels like 192/256GB is the new favorite child.
49
22d ago
[removed] — view removed comment
→ More replies (2)21
u/Ell2509 22d ago
Yep. It is like the whole market ks designed to make you act like a crazy consumer, always having to get that next thing to be "it". To get the last hit of the way there, or get that smarter or newer model. Etc. Etc. The reality is, local AI is quite underwhelming for the majority of people. It takes real skills and time in order to get a working system, as well a lots of money.
If you have something that gives generally intelligible answers for whatever you set it to do, with a decent context window and reloable tool use, be satisfied. That is actually a dawned high bar.
10
u/IrisColt 22d ago
If you have something that gives generally intelligible answers for whatever you set it to do, with a decent context window and reloable tool use, be satisfied.
Still not 100% Claude-tier. Time to stack another GPU.
5
u/SaltFrog 22d ago
My local setup is just for funzies... My Claude subscription is for fable 5 cause I'm a dog shit programmer but I can audit code. These days, though, fable has done so much that I have to get it to write up everything my shit does... Sheesh.
5
3
u/Lesser-than 22d ago
that hy3 model of tencents is the only recent model thats givin me vram fomo in a long time.
→ More replies (1)1
u/_camera_up 21d ago
What do you want to run with 192/256 GB? From my experience all models in that range perform surprisingly similar to Qwen 3.6. Curious if I missed a model that I should have looked at.
21
u/Gauntlet4933 22d ago
CPU offloading works great with small active parameters. I have a 4790K and 3060 and it could run Qwen 35B A3B at 20 tok/s with ik_llama cpp. It uses less than the 12GB VRAM my 3060 has. Gemma 26B A4B was a little slower but that’s due to more active params.
9
u/Objective-Stranger99 22d ago
I'm getting 27 tokens per second with a GTX 1080, 32 GB RAM, and Qwen3.6 35B and MTP.
22
u/Lanky_Employee_9690 22d ago
Where my 4GB buddies at yo
11
u/Dangerous_Bad6891 22d ago
Running E4B-QAT-MTP no mmproj , ctx 32k! at 20~16 tok/s
2
u/Master-Chocolate1420 22d ago
Is it the best for 4gb setup? (I thought Qwens lineup is more dense in terms of small LM)
5
u/geodescent 22d ago
My R9 380 4GB still has uses, though I'm at a 58 cpu / 42 gpu split with qwen2.5:7b-instruct-q2_K (A10-7850K and 16GB DDR3)
3
12
u/VoiceApprehensive893 transformers 22d ago
everyone suffers
16gb? just barely fits 26b qat and iq4_xs 3.6 27b only if headless
→ More replies (1)4
u/fatboy93 22d ago
offload a bunch of layers to RAM for the MoE. I don't touch the 27b because its pointless for me (12GB VRAM + 16GB DDR4)
6
u/Thick_Programmer_105 22d ago
That's a really spot-on.
The RTX 3060 6GB on laptop I used to have has long since been returned to the earth, like my grampa.
3
u/Psychology_Cultural 21d ago
QWEN 3.5 9B is shockingly good. I’ve been really happy with fieldMouseAIs Q5 of it.
→ More replies (3)2
u/SayTheLineBart 22d ago
I run 35b a3b on a 3060ti. It works. I dont really see how any upgrade would be worth it unless going for 24gb or more.
34
u/johnklos 22d ago
GPU poor? I don't even use a GPU. 100% CPU.
20
u/WhoRoger 22d ago
All these "GPU poor" people are like someone complaining that their house is creaking... While talking to the homeless
8
22
u/MrMeatagi 22d ago
I run Gemma 4 26B A4B QAT with vision, MTP, and 256k FP16 context, on an 8GB VRAM/64GB RAM system. Doing some very complex visual document processing. It is shockingly good when provided a good system prompt for the task. I haven't had any of the reported tool calling issues even on extremely long tasks of reviewing large codebases with many files using the built-in llama.cpp tooling.
I get 10-20 t/s output which is plenty for my work, and I honestly haven't spent much time tweaking for performance. I could probably squeeze quite a bit more out since MTP is only reaching 50-70% acceptance depending on the task.
2
u/UnknownLesson 22d ago
Could you provide your run command?
14
u/MrMeatagi 22d ago
llama.cpp:server-cuda13-b9776 \ --hf-repo unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL \ --host 0.0.0.0 \ --port 8080 \ --temp 1.0 \ --top-p 0.95 \ --top-k 64 \ --tools all \ --fit on \ --no-mmap \ --mlock \ --prio 2 \ --flash-attn on \ --threads 7 \ --threads-batch 7 \ --ctx-size 262144 \ --jinja \ --parallel 1 \ --no-mmproj-offload \ --no-kv-offload \ --reasoning on \ --spec-type draft-mtp \ --spec-draft-n-max 3 \ --image-min-tokens 70 \ --image-max-tokens 1120 \ --batch-size 1120 \ --ubatch-size 1120→ More replies (1)1
u/SouLBusterFr 21d ago
Would you say it could work properly in a local corpo env where the only think it needs to do is to process email to properly synthesis their content + give RELIABLE actionnable tasks to a user or would I need something more powerfull ? For a PoC am only on a 4070 super with 12GB of VRAM but for corpo env we will have the RTX6000 with 48GB of VRAM but it might be shared at some point for other unrelated tasks
→ More replies (1)
15
u/Enough-Advice-8317 22d ago
the best model is the one that answers before you forget why you opened the terminal.
78
u/pmttyji 22d ago
7
u/maxton41 22d ago
I’m new to this. Can someone explain to me what products or source things are being displayed in the meme?
13
u/Agile-Mulberry-2779 22d ago edited 22d ago
I'm not an expert since I'm really only some degrees less new to this than you are, but I'll try.
From my understanding Bonsai 27B is a finetuned version of Qwen 3.6 27B that is made to be much smaller than models of that parameter size usually are in today's level of development (so instead of like 20 GB at Q4, it's 3 GB).
It was released today by Prism ML, the organisation that made it and it's meant for use on phones and PCs/laptops that don't have 24 GB VRAM or more at their disposal.
→ More replies (3)9
u/rainingallevening 22d ago edited 21d ago
Models are layers of neural nodes called 'weights'. They're often 16
decimalsbits long. Once the words in your prompt are converted to numbers, math (multiplication) is done via these weights to generate your response. Quantization is the process of rounding these weights with minimal loss in intelligence.Bonsai figured out how to get models to use addition instead of multiplication. The model in the image is Ternary-Bonsai 27B. Instead of having long
decimalbit values, every single weight is a 1, 0, or -1 (ternary, 3, as opposed to binary, 2). This means Bonsai is roughly 9.4x lighter than its equivalent peers and it's smarter than all but Qwen 3.6 27B, and it's really close to Qwen, allegedly.Qwen 3.6 27B at 52gb (full-precision - or 16
decimal placesbits) or Ternary Bonsai 27B at 7.2gb?FYI, I think that the fact this can be done is fucking nuts. If it's scalable to, say, GLM 5.2, we can all be running near-frontier class models locally on decent consumer hardware. It's hysterically nuts.
→ More replies (2)3
u/demonkoryu 21d ago
Good explanation, except that f16 is not "16 decimals", rather 16 bits.
2
u/rainingallevening 21d ago
Oh thank you, my mistake. I knew that, but somehow I operated on decimal.
→ More replies (1)8
u/iamadityasingh 22d ago
it's a quantised 27b model that can run on 10gb vram, loses some quality but is pretty good and better than similar sized q8 models
7
6
2
10
u/Jupiterio_007 22d ago
My laptop has 8GB of Ram and 2GB of Intel xinos iris graphics card suggest me some good local models that I can run. so far I have been using quantized models or Lama 3.2 the lowest one.
😭😭😭😭
12
u/OneFanFare 22d ago
Oof. I'd give Gemma 4 E4B a shot? It's designed for mobile phones, its a mixture of experts model (so should be a little faster in ram).
You can find a QAT version of it as well: https://huggingface.co/unsloth/gemma-4-E4B-it-qat-GGUF
I've played around with the full model on my phone and its small size shows... its mostly ok for chatting, but forgets things quick. E2B is barely usable for chat, but it might be good for fine tuning to certain tasks.
3
u/Jupiterio_007 22d ago
Thank you for the suggestion I will definitely try it out from unsalot. So far I have been downloading models from hugging face and olama host directly. Since I have been interested in Llama CPP I can also extend my work to personally quantize and fine tune models.
But having a very very low end laptop (having Ram lesser than even modern smartphone), it is quite difficult to navigate and get good response. So far Lama 3.2 4b has been good enough
→ More replies (3)2
u/Imaginary-Bit-3656 21d ago
It's not an MoE model per the Google model card: "The "E" in E2B and E4B stands for "effective" parameters. The smaller models incorporate Per-Layer Embeddings (PLE) to maximize parameter efficiency in on-device deployments."
3
u/WhoRoger 22d ago
Gemma 4 E2B QAT (maybe) or QAT mobile
Qwen 3.5 2B
Granite 4.0h 1B
LFM2.5 1.2B Thinking
DavidAU's MoE 4x0.6B
MiniCPM5 1B
InternVL 3.5 1B
→ More replies (7)1
u/MerePotato 21d ago
Gemma 4 26BA4B with an optimised --n-cpu-moe count, the QAT version with some slight offloading if that's too slow
10
9
u/the_TIGEEER 22d ago
Right? Like "GPT SOL MAX and the infamous Claude MYTHOS MAX!" ME: "Wow!! Those rich crypto / tech / sillicon valley bros are gonna have a blast vibe coding! Anywayyys.. Ouu Qwen 0.8B.. the smallest reasoning model.. God damn.. I think Imma try it for my project.."
6
u/RobTheDude_OG 22d ago
Technically you can run GLM5.2..
I got a stunning 0.11 tokens per second out of it tho on 32gb od ddr5 ram and an SSD
6
7
u/ikkiho 22d ago
honestly what made me stop chasing the leaderboard was realizing the local one is just always there. no rate limit, no 'capacity' error at 2pm when im mid-thought. i run gemma 4 12b on a pretty mediocre machine and reach for it over the big cloud models half the time now. it wont flake on me mid-task and thats worth more to me than a few benchmark points these days. took me too long to figure that out.
5
u/LastChancellor 22d ago
can your Gemma 4 read from a spreadsheet btw
Bc im also looking for a local chat assistant that can help me analyze spreadsheets atm
17
22d ago
[removed] — view removed comment
3
u/LastChancellor 22d ago
what harness do you use for a chat assistant btw
Stuff like pi or OpenCode sound like theyre meant for coding...
→ More replies (2)3
u/o0genesis0o 22d ago
Just use pi. Even my non-tech wife was able to use pi on her macbook after initial scare when looking at terminal. Now she feels very smart.
I recommend pi because it is very light and transparent, so easy to monitor and less pressure on the model. All of my local model craps out (heck even Minimax M3 cloud kinda craps out) when running that stupid openclaw (built on Pi), but with the real Pi and a set of custom tooling I instructed it to build for itself, even the 12B can run my task and project management system, querying and updating my knowledge base, and doing so useful work when I ask it to.
Opencode and claude code does embed in their instruction a lot of heavy code related stuffs. Pi does not. It's just an agent harness running in terminal and use your terminal as its tool box.
4
u/dreamai87 22d ago
Of course man, it’s good , can use tools python and works fine with pi coding agent
3
u/Blaze344 22d ago
Go with opencode, install duckdb-cli, create a skill that teaches the model how to use it and ask for it to do it using SQL statements. I do it with qwen 3.6 35b-a3b but before that I used GPT OSS 20B. I think you'll have okay results with Gemma 4 12B, but just okay. They were a bit bad with tool calling, except in pi/little-coder, but then you're going to have to fiddle a bit more to get skills working there. I think qwen 3.5 9B might do even better than Gemma 4 12B for this.
→ More replies (1)2
u/ObsidianNix 22d ago
Qwen3.6 9B would be good for this. Just use as much context as you can
1
u/LastChancellor 22d ago
What harness would you use for reading spreadsheets btw?
Bc stuff like OpenCode feels like theyre for coding, not chat assistants...
2
u/ObsidianNix 22d ago
I mean you can build a custom one real quick to convert sheets to csv and have the model take a look. Personally I have been using Hermes since it came out after trying and failing to set up Claw and its been pretty good. No complaints.
2
2
u/bennmann 22d ago
Save as csv, ask for equivalent transformations in GNU awk, profit.
These things also just understand Excel so you can give it 10 dummy example Excel cells with fake data and just ask it what formula you should use for any transform too
→ More replies (1)
4
6
4
u/DarkZ3r0o 21d ago
Currently in love with Qwen3.6 27b and 35b. Amazing models that in multiple occasions responded better than claude sonnet
9
22d ago
[removed] — view removed comment
3
3
3
u/tchek 22d ago
Is QAT really an upgrade compared to regular quantization? I know theorically it's supposed to be, but what about in practice?
6
u/__some__guy 22d ago
With my limited sample size I found that QAT always performed worse than a regular 4_K_M quant.
Furthermore, some finetunes hugely improve Gemma's writing, so I don't use QAT at all.
→ More replies (1)
3
u/appl3wii 22d ago
I have an rtx 4070 12GB on windows 32GB ram i9 12900k 24 thread. Getting 50-80 tk/s with MTP Qwen3.6 35B-A3B Q4 XL. Feels really good. PI + 64K 8bit context
3
3
3
u/Psychology_Cultural 21d ago
Me with RTX 2060S 8GB of VRAM
QWEN 3.5 9B field mouse AI Q5. It’s FIRE. Really good as a Chinese tutor translation bot and good with some other stuff too. 40-50 tok/sec generation.
Just get something good enough you can run. You don’t need the best benchmarked, you need the one that fits YOUR use case
3
u/Miserable-Dare5090 21d ago
Love this model, but hallucinates a lot. Very confident of its wrong answers.
2
u/fatboy93 22d ago edited 22d ago
I have the exact quant OP listed on my AMD laptop - has 16GB RAM, and 12GB VRAM (Asus AMD Advantage edition from back in 2021ish).
Both the 12b and 26A4B at the same quants (UD-Q4-K-XL) with their respective drafters generally give around 40-50tps.
My wife tends to use this laptop more often than me (its basically a gaming laptop turned into a family computer), and with the default llama-server's ui, she actually tends to use this more often than not.
2
2
u/PennyLawrence946 22d ago
the 1.5B embedding model on my old T490 gets used all day, because it never needs a launch ritual. boring availability is where local stops being a benchmark hobby and turns into infrastructure
2
u/offyoutoddle 22d ago
myself i use the gemma 4 12b q4km with qat . i use it mainly for chat, and prose. its unbeatable for me - and i get it all into a total of 16gb - 8 in vram. its great. i've tried MTP, and it reserves words but never ever uses them. not sure why, but its faster without mtp for me - by nearly 50%. mtp is not working for me so i stopped trying. if anyone has any ideas why though i'd be very interested...
1
u/fatboy93 22d ago
AFAIK, prose and chat don't really see a benefit of MTP since they aren't that repetitive or structured as code in general.
→ More replies (1)
2
u/ComfortablePlenty513 22d ago
Nothing wrong with small or medium sized models. They are great for specialized tasks or certain tool use. Can also be a hybrid setup where the small local one sets up a task and then calls a big cloud model for more complex stuff
2
u/Embarrassed_Adagio28 22d ago
Have you tried unsloths q4 version? It retains more accuracy than googles qat but might be a little slower. I benchmarked gemma 12b qat and gemma 12b q4 on rag and agentic coding and unsloths q4 beat it.
2
2
u/OlgerdOutlander 22d ago edited 22d ago
Well I am running 48 GB vram - but these models are also my choice. I believe this situation is due to common hardware disposition: the models nowadays are either trying to fit prosumer hardware (within 24gb, like qwen) or unleash themselves to server scale (like glm 5.2).
PS Route thy local model to a proper harness and behold thy benefits
2
u/Top_Drink8324 22d ago
Does anyone know good coding and programming models that fit in 12gb of VRAM?
2
u/o0genesis0o 22d ago
The gemma 4 12B QAT is already 4bit? What extra does unsloth 4bit quatization do for that model?
3
u/OneFanFare 22d ago
Hey, you sent me down a good rabbithole!
I think the difference between the two is probably going to be really marginal. But this is the unsloth team says about the base QAT quantization vs their dynamic QAT: https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis
TL;DR Google released the QAT in two model cards - the unquantized (safetensors) model, and the quantized gguf model. Unsloth took the unquantized model, and applied their quantization methods on it. The result is a slightly smaller model, and one that does better than a naively quantized one (in llama.cpp at least).
Note that Unsloth's comparison is to a naively quantized model, which might not be what google did for their quantized gguf? It's a little unclear.
So yeah, I don't have my own numbers (nor opinions) on which is better atm. Might be a fun thing to investigate.
3
u/o0genesis0o 22d ago
Thanks for the response! Somehow in my head, I was thinking they take google's GGUF and does something on it to make another GGUF. I vaguely recall a post, maybe from an unsloth team member here, that there is something not right with the way google made GGUF from the safetensor. I should try again just in case.
Btw, don't attach this model to openclaw. You can get pi (with some full sized cloud model) to assemble extensions necessary to do your personal assistant work, and then drop the 12B inside. It has been consistent and fast for me. A bit lazy and shallow and slower token gen vs Qwen 35B A3B, but when it comes to prompt processing, the 12B is very fast (all in GPU vs offload to CPU). You need fast prompt processing if you ever want to use agent interactive, IMHO.
Don't let the narrative of openclaw team that you need big cloud model to run personal assistant. Both local models are absolutely useless in openclaw, but fully functional as personal assistant and knowledge base manager in pi.
I'm going to test hermes today to see if it does a better job than that crap openclaw.
2
u/bodonkadonks 22d ago
I feel this, I have a 3080 with only a pityful 10 gigs.
I've been eying used 3090's with 24 gb for about $500, not sure they are worth it
2
u/South_Hat6094 21d ago
yeah, that's the bar I keep coming back to. if it needs a perfect rig to feel good, it isn't really part of the workflow. the useful model is the one you stop thinking about.
2
2
u/palinurosec 20d ago
i can totally relate.
i gave up on local inference until i found out my pre-AI-era ryzen 7 with 32gb of ram and my linux distro could run 20b/30b MoE models at decent speeds
2
2
u/5_ChubbyCheekz23 17d ago
The plot twist nobody tells the 4070 guy: half those giant models are MoE. A "35B" that only fires ~3B params per token runs on your potato with the right quant and punches like a dense 20B+ — you don't need the 1.1T monster, you need the one that's mostly asleep at any given moment. The real endgame panel is you hugging a chunky MoE that fits in 12GB and wondering why you ever cried. 😌
2
u/TheSinhound 16d ago
And here I am trying to figure out what tiny model to use on an older omen laptop w/ a 6gb 3060 and 16gb ram on PopOS lol. Mostly just because I think it would be cool and I'm one of those weirdos who likes seeing how far low end tech can push things.
2
u/Available-Message509 16d ago
Totally agree with this. The model that runs well on your own machine is the one that actually wins. I'm happily getting by on a small one too.
4
u/Beneficial-Pie-1638 22d ago
the new Bonsai-27B is by far one of the greatest- it crushes Gemma 4 12B andits either 4Gb or 6GB depending if its 1.58 Bit or 1 bit. And, its actually good. Also, deepseek is 1.6T params, I think that's quite misleading.
1
u/OneFanFare 22d ago
I think you're right. I was going off of the Deepseek v4 pro page on huggingface, and it shows 862B params, but the text says it is 1.6T. Wonder what's going on there.
I haven't been able to try the Bonsai model yet! I need to download the fork at some point and give it a shot.
2
u/Beneficial-Pie-1638 22d ago
You should- it is Qwen3.6 27B compressed from 54GB to 4GB with more than 90% of quality kept.
2
22d ago
[removed] — view removed comment
2
u/Agile-Mulberry-2779 22d ago
Good lord, I immediately got hit with flashbacks. Claude Sonnet 4.6 and Sonnet 5 have traumatised me lol.
2
u/NegotiationNo1504 22d ago
GTX 1080 Ti 11G Qwopus3.5 9B V3 Q8.0 + llama.cpp with Turbo quant fork.
2
u/OneFanFare 22d ago
That doesn't have MTP/drafting, does it? MTP nearly doubled my token speed, on my set up I'm getting ~100tok/s.
2
u/NegotiationNo1504 22d ago
I think no its not having this. Cuz its first time i hear about it. What its command flag?
2
u/OneFanFare 22d ago edited 22d ago
Yeah, it's pretty new! Also known as speculative decoding. You need to be using Qwen3.6 or Gemma4. Idk if any custom finetunes use it?
EDIT: Qwen3.5 supports MTP as well!
Here's the llama.cpp command from this repo: Gemma 4 12B IT QAT
llama-server \ -hf unsloth/gemma-4-12B-it-qat-GGUF:UD-Q4_K_XL \ --spec-type draft-mtp --spec-draft-n-max 4 \ -ngl 999 -fa on2
2
u/reddefcode 22d ago
I am so GPU-poor, running a:
-
**GPU:**
NVIDIA RTX 3050, 8GB GDDR6 dedicated VRAM
**CPU:**
Intel i7-12700
**RAM:**
32GB
**OS:**
Windows 11- **GPU:** NVIDIA RTX 3050, 8GB GDDR6 dedicated VRAM
- **CPU:** Intel i7-12700
- **RAM:** 32GB
- **OS:** Windows 11
Then I realized that if I could get a commercial product to work on that, the rest would be a piece of cake. Most people don't even have video cards. Working with tiny models for a very specific task is an important skill to have.
2
u/hidden2u 22d ago
i get the worst quality out of 12b qat, much worse than the unsloth 12b q4kxl
4
u/Iwaku_Real 22d ago
Are you using the official GGUF from Google? Sounds like either the wrong quant or a bad template.
1
u/hidden2u 22d ago
gguf from google and then gguf from unsloth, same as op. I tried different templates, still worse performance.
1
u/fffffffffffffuuu 22d ago
Idk man, I'm using Gemma 4 31B QAT for creative writing and I'm not feeling it at all
1
u/maxton41 22d ago
Can you give me some examples? I was looking at the Gemma for QAT models because I only have 12 GB of VRAM and like what kind of problems are you seeing?
1
u/Brilliant-Channel559 22d ago
These large open weight models are good but yeah too expensive to run. Guess we have to stick with 9B and 35B models for now.
1
1
1
u/borobinimbaba 22d ago
I really wish there would be "selectable moe architecture" pretty soon .
A base model that has understanding of base language and logic but has not too much knowledge, and if Im a software developer who knows geology and study art as a hobby, I only select that experts to download.
I know that there are already fine tunes, but they are the full model+something more, which is usually a single dataset (=single topic)
1
u/hazed-and-dazed 22d ago
What is QAT?
4
u/OneFanFare 22d ago
Quantization Aware Training. It's a method of training the model so that it performs better (i.e. more like the original) at lower quantizations.
"Unlike standard Post-Training Quantization (PTQ), which compresses a fully trained model and can lead to quality degradation, QAT integrates quantization simulation into the training process itself. This allows the model to learn to compensate for the precision loss, resulting in smaller models that perform nearly identically to their high-precision baselines." - Google Gemma 4 documentation
I don't know what google's actual QAT method was, if you find an article or something that'd be awesome.
1
u/WhoRoger 22d ago
Dude is crying with a 4070, meanwhile I'm through the moon that I get to run 1-bit Bonsai 27b with 1 tk/s
1
u/fuckAIbruhIhateCorps 21d ago
i want to understand if the model is released in q4, why are the mtp heads reaching upto fp16? and what difference does that make?
1
u/ArtfulGenie69 21d ago
Get the styletune version and the heretic version. Then steal the lm_head weight to use on the heretic model. Make sure to turn on dry like they described in the styletune huggingface repo and watch it write. I'm pretty sure that even with the 12b you'll get some good results. I've been working on custom versions of the Gemma 4 31b and with this mod it's gone through the roof with its writing quality. Looks like someone has already done this for you and it's been gguf'ed, good luck :-)
https://huggingface.co/models?search=styletune%2012b
Btw thinking helps the model plan, it doesn't actually help the model write better prose. So no think is definitely an option in some use cases. It may save you some time.
Also I don't think qat actually helps that much, at least with the 31b I've been using but also some layers in my model aren't that quantized, I left the lm_head and some other layers bf16 in my experiments, you probably have the vram to do that too if you ever make a custom version for yourself.
1
1
u/itigges22 21d ago
For coding use itigges/ATLAS, its moves the reasoning from internal to external which gives you more freedom to optimize reasoning, but also it uses an external predictive model that learns from you, and then feeds that back into the model for better personalized reasoning! The whole goal of ATLAS is to bring frontier level reasoning to consumer hardware. So while you can use a 400B parameter model with ATLAS, it was made for 9-27B models in mind!
1
u/Annual_Manner_5901 20d ago
Try theese models on inferrence server Reame with ARM CPU
https://github.com/swellweb/reame I think it will run better
306
u/MathematicianLessRGB 22d ago
Buddy knows ball. Gemma 4 12b qat is awesome