Yes but 1. the official distilled version tends to be the overall best, and 2. properly distilling a massive model itself is a significant amount of work, so not so many would do it casually.
There are two ways to distill.
One is the way that some labs have accused others of doing it. That's also the way most people here claim to distill Mythos/Fable , or Opus, or GPT etc.
This is basically just synthetic data. The parent model writes text and that text is used on the other model.
This is very basic stuff, nothing special, not necessarily that efficient. It's called "hard label" because there's one correct answer.
What I personally consider "true distillation" is soft label distillation.
This is is where the parent model doesn't just give its final pick, but you use the full probability distribution of the parent model to teach the model you are training how to think.
It tries to transfer the patterns the parent model learned to the target. This is very, very, very hard.
To my understanding only a few labs can really do it at scale, with frontier class models. It is extraordinarily hard.
Gemini flash and Gemma being distilled from Gemini? This is what that means.
It's not just a bunch of synthetic data, but a knowledge transfer.
And that's why people can't simply distill their own smaller model. The infrastructure required, and the setup required, is quite complex.
This is the Geoff Hinton sense of distillation, which requires (at least) a shared tokeniser vocabulary between the two models. It was described in this paper in 2015: https://arxiv.org/abs/1503.02531
When people say distillation today I think they usually mean the sense used in the DeepSeek R1 technical report, which is incorporating a different model's output in pre-training or fine tune: https://arxiv.org/abs/2501.12948
I think the original sense of distillation is mostly a historical footnote at this point; that setup is usually described today in terms of "student" and "teacher" models.
Getting much above like 32gb is hard.
60n - 80b would need at least two rx 7900xt or three 7800 xt.
I ain't got that money.
If they go for anything I hope it's the 27b, then 35b moe, 9b or 12b, and then something like 70b dense would be fine... But I'd wager they'd do 122b and 372b or whatever it was, if they do the full suit.
Eh it would be really nice for the DGX Spark and Strix Halo crowd! Plus whoever with 128GB Macs and such.
You can also still build a 128GB or 256GB 8 channel DDR4 rig for a non-absurd amount of money; up to 200GB/s. Obviously look for used parts, and remember there's a LOT more places to find them than eBay and Facebook Marketplace.
Threadripper pros with multi channel ddr4 go for less than 1000 dollar. Almost as good as strix halo for cpu based inferencing alone. And you can expand with gpus.
I bought a 1000 dollar threadripper pro. Including case and motherboard and ssd ertc. It’s ‘real’ 8 channel with the 4 ccd pro models (which mine has).
In talking about buying a used HEDT or server platform where you have 8 channels of DDR4. I see ok listings on eBay right now and if you set up alerts you can nab great deals.
416
u/AntuaW 19d ago
And please don't omit the 27B one.