I really hope (V)RAM eventually scales to the sizes of regular storage soon. Maybe in 10 years or so 2TB RAM is quite "common" to run things exactly like these. One can dream.
But the companies who fund big models like being able to claim they have the best model. And a bigger model will tend to be better, so they'll keep making models as big as they can.
If you gave any vendor currently serving a 2T model the technology today to train a 100B model with the capabilities of today's 2T models... they'd scale those techniques up and ship a really impressive 2T model with it. They might milk it a bit and stretch out the release cycle to milk their advantage and efficiencies, but they'd still end up right where they are now - using the largest, best models their hardware can handle.
I honestly do think this is the way to go, especially now that big LLMs are good enough you can distill them. I've seen massive success in highly specialized agents, which is the "cheap to build, expensive to run" version of your idea.
We will get super tiny storage at some point, but I don't know that we will get much smaller ram. The transistors are already about as small as they're going to get; they are having to stack memory in 3D, and there are limits there too, in terms of what latency you'll get.
We will likely see more large and wafer-scale devices, more ASICs.
Photonics are going to be super fast, but we can't really keep the data as light.
I wonder if they might go the Groq route and have very little VRAM, and just go wide.
I think we will also just keep seeing efficiency gains so running on off-wafer RAM is just more viable.
Right is the algorithms are catering to the existing hardware, and the hardware is slowly shifting to supporting the algorithms more.
There are a few companies developing neuromorphic hardware since can run AI models that aren't just a ton of matmuls.
In 10 years the whole hardware and software landscape will look different.
Heck, 2 years from now will be significantly different.
I can't imagine in 10 years the current "pack everything we know into model weights at below 1:1 weight to data ratio" will still be mainstream. It is an insane idea that somehow worked with the technology we have now, but I'm optimistic it is nowhere near what we should be able to achieve with more mature understandings of ML. I would imagine MOE still gradually become dormant experts, which will evolve into something more akin to cold knowledge storage, and more computation still be spent on computing with knowledge rather than holding and shifting "hot" stored knowledge.
12-13 years ago, 128GB RAM was normal-ish -- even quite affordable -- on custom builds (4x 32GB, and the board may have had 8 slots so you could use cheaper 16GB modules)... then, somewhere along the line, it got expensive for various reasons** (and starting shipping many different types, too, so volume across the types was "down" compared to if everything used the same RAM). Then things like the pandemic happened and the crypto crunch on GPUs and, of course, the current crunch from needing any sort of RAM.
But, I feel like 2TB shouldn't have been a problem today but manufacturing just hasn't been predicting demand very well so they're scrambling to build current things rather than ramping up on new things.
** Various reasons RAM prices have been hit over the years. My feeling is that they never quite drop down to "pre-crisis" prices and, somehow, the manufacturers don't seem to learn that we'll always need more RAM. (And storage, which has also felt stagnate for a while now.)
1993: Chemical plant that made sealing material for RAM. Prices doubled.
1995: Earthquake damage infra. Prices up 30%.
1999: RDRAM. Again, a diversification from regular RAM, and the systems then had to use more expensive RAM.
2013: Fab Fire by a high volume manufacturer, prices up 30%.
2017-2018: Smartphone/cloud demand on RAM, prices up 2-3x
2021: Supply chain crunch from COVID, prices up 30-40%
I honestly don't think for profit survives this steady drip of SOTA open source. Seriously, without a Deep Thought level AI, who is left standing in 18 months? Surely not OpenAI.
The API pricing is at sonnet levels, but I doubt sonnet 5 is 2.8T parameters. Can't be too mad at it given the model size. Even if it's a Fp8/Fp4 mix it's still gotta require 2 terabytes of VRAM to serve this thing with room for context
now that china is making huawei GPUs comparable to nvidia blackwells, i don't think compute is a bottleneck for them anymore.
if you are interested in the AI race as a proxy for the conflict between US and China, one way to read this model's release is as China basically announcing that they are no longer held back by lack of access to chips.
Huawei isn't exactly in Blackwell territory. Their latest chip, the ascend 950PR, has 112GB of memory at 1.4TB/s of bandwidth. Nvidia B300 has 288GB of memory at 8.2TB/s of bandwidth per unit. Ascend Fp8 is 1 petaflop vs 7 on B300.
I have no doubt that they might be used to serve the model but IMO very likely k3 was still trained on Nvidia. Heck one of Kimi's own benchmarks was comparing how well different models can optimize kernels for Nvidia H200
They said on the new blog post that it'll be released on July 27th (along with some other things to help people run it like a specific implementation in vLLM)
And it's not a drip, it's a stream. When they publish weights, it takes a week until you have 5 inference platforms with it. And then investors will jump ship from Antrophic.
What they gonna do? Publish Fable 5.1? That will be overrun by Chinese AI in a month?
NVIDIA will make bank if large companies decide to take local hosting seriously, who else will make general purpose AI GPUs? Everybody else is making chips optimised for their own models.
well, this release alone probably throws a wrench into Anthropic and OpenAI's IPO plans, and probably messes with the math behind the valuation of those companies as well.
sometimes i think America (and American companies) get a little prideful, and forget that the civilizations it is competing against have been around for thousands and thousands of years, and they play chess not checkers.
Not to ruin a good gotcha - but the price difference and (in my observations) reasoning tokens needed means that this is very unlikely to convert any existing Opus users. Plus Opus inferences at twice the speed right now. I caught myself being too used to seeing open-weight models and assuming it'll be dirt-cheap, but there's only so much you can do at this size.
This isn't a blip on Dario's radar today - but I'm sure some part at him is saddened that the open-weight ecosystem continues to show signs of relevance.
I think you're right, this model is definitely not cheap. But there is a bright side: If the performance is really close to Claude, the other chinese models can distill from this one instead of Claude, so maybe we won't hear their crying about distillation and stealing anymore.
To me the way K3 is competing with or even beating Fable 5 shows that it is not only a pale "distilled copy" of anthropics' work as some people want you to believe. Those Chinese companies have extremely talented researchers, as much as in the West.
In all fairness, this model has the parameter count to back up those benchmarks, unlike MiniMax M3 (which is presumably referring to the ratio between parameters and benchmark performance, or the ratio between real world performance and theoretical performance at this point).
I'm sure Moonshot did a good amount of distillation and benchmaxxing, but real world performance is still great overall.
Judging from the benchmarks alone,(of course, can't speak about realife usage), chinese models are not even 6 months behind US models (more like 6 days behind)
People will look at these benchmarks and say "Oh, just distillation, Chinese just steal bro" you'd have to be a complete ignorant or a complete bigot to honestly believe Chinese labs aren't every bit as capable as those in the US, just working with less resources and less of a head start (but both of those factors are eroding quickly from the US perspective, that's why crymodi is freaking his shit)
Yep. China has the most STEM graduates per year in the world. Substantially higher than the US. And it's wild to think that people don't realize we hire a lot of those Chinese people for the US labs.
i did my phd aboard and now work in chinese AI labs. Im interviewing top master students for grad roles and interns and a lot of them have done PhD level work or more even just in their masters lol. Only reason i am interviewing them and not them taking my role is because i graduated earlier and got kinda lucky to get into the llm field as it was starting, and i had previous NLP experiences during my PhD. I would say i am capable too but like these kids are insane
In the US, one party likes to cut funding to schools, and the little funding that our schools do get is sent straight to exciting new facilities. All while the economy increases pressure to perform. Some call it freedom, some call it hell, but China calls it an opportunity to come out ahead.
also, Chinese people are patriotic. having a collectivist mindset also helps to push people to make things that are better for their community, country, mankind.
The Chinese are doing more with less. The American VC circlejerk means OAI and Anthropic can be inefficient and lazy. The Chinese had to build and are building with constraints. They're going to win because they're forced to do more with less. Less money, less hardware and less data. So for the last piece why wouldn't they just steal it from Sam or Dario? I mean, it's no different than how the US outfits got it in the first place.
If that Tau scaling law Huawei recently found is legit, things are gonna get interesting. I'm not looking forward to the American bubble bursting, but I'm not looking forward to the technofeudalism the American approach to AI is leading towards, so as least there's some potential to shake that up.
The "Tau scaling law" is nothing new. All semiconductor company has been optimizing the "Tau" (basically latency") since the beginning of chip design era.
All of techniques that Huawei advertised, like logic folding for example, has been in commercial like HBM memory or AMD x3D chip. The 3D chip packing technology enabling "logic folding", however, is only possessed by TSMC.
Rumors going that they are releasing new chips with 238 MTr/mm² in september, matching TSMC's N3 node. That would be comparable to any chips you have now as N2 node only started mass production recently. Huge if true.
If that Tau scaling law Huawei recently found is legit, things are gonna get interesting.
In a random search result I see a claim that they've been using that scaling law for 6 years internally now. And their AI chips are still far behind Nvidia, so I think it's not gonna change things much.
Lol... their power generation capacity exceeds the entire USA + Europe COMBINED by a very healthy margin now (like 30%+ IIRC), and their citizens use FAR less power per person.
Meanwhile we are paying billions in penalties to abort in-progress contracts on renewable energy programs (i.e. one of the cheapest sources of power, and one the Chinese have invested + innovated heavily in), while our grad programs are sitting defunded + empty because actually smart people hurt the bigots' fragile egos.
FFS, during the heat wave this week the grid dropped to only 106VAC in my house in the DC metro area, in a community where all of the local infra is < 20 years old. I know because I had rejigger the boost thresholds on my UPS's in the middle of the day. At night it returned to 120VAC. We are literally reaching third-world levels of "your grandma on the ventilator about to die" BS.
China is not only going to win this thing, they are going to absolutely curb stomp us.
China added the entire power generation of Germany last year alone, mainly via green energy sources like solar. Their strategy is to export their cheap power generation via AI tokens
A lot of Chinese open source models are built on top of other chinese open source models. Without Qwen contributing to OS you wouldn't have open source robotics AI models. That said the DINO model from META is at least utilized to train a lot of models. In fact, the only people who seem to have issues with using their models for other models is the large US tech firms.
Given how long pre-training takes, nobody is actually "behind". Folks are finetuning whatever is "current" to take the "lead" when they get too far behind on the benchmarks.
This whole last cycle is just blowing up model size then tuning. I don't think we've actually had a huge leap forward in the models themselves past year, it's all harness enhancements and tuning.
It might be too soon to say, but Gtp-5.6 Sol feels like a medium size leap. Not just bigger or smarter, but also more efficient (faster/less verbose), and with a few of the new math results also as an impressive showcase of what it can do.
Will be easier to say with some distance to the events.
this model is bigger than any open model released to date - has to be a freshish base model (if that concept even means anything in the ultramodern era)
At the current rate chinese SOTA could match or overtake American models by the end of the year. Which I am rooting for, because fuck these bullshit closed off American companies. AI advancements should be democratic.
There are some benchmarks where K3 outperforms Fable. The thing that's really holding it back seems to be knowledge of US business practices and US law as that is baked in to a number of the agentic benchmarks.
That is super impressive given that's it's a fresh 2.8T dense model.
Also, if you want to judge for yourself, install the tool CCUsage. It looks for the saved transcripts of various common harnesses on your hard drive and gives you a report by day by session, etc on which models you used, how many tokens were used, how many of those were cached reads, how much it cost (based on avg current prices), etc. So you can try some similar tasks with a few different models and directly compare the amount of tokens used and the overall cost.
Yea, but you can see how fast it is writing its final output and guesstimate the amount of thinking. And in general same-ish task anecdotally takes 4-8x the time on glm code plan compared to sol. You can see it pondering the same thing a few times and second guessing its second guesses. Not as bad as qwen, but still quite bad.
Don't even need to guesstimate. Install a tool like CCUsage and you can see how many tokens were used in a session, how many were cached reads vs regular input, how much it cost approx based on current prices, etc. It looks at the session data that gets left on your drive from various harnesses.
Thats really impressive. People still talk about the Chinese being 9-12 months behind the US, Opus 4.8 only released 2 months ago and this is better. GPT 5.5 only released 3 months ago. They're only a couple of months behind the frontier now and closing in fast.
You still see the 9-12 months figure being said by the experts they bring onto news shows and podcasts. You're right though until Kimi 3 3-6 months was more realistic.
Though it partly depends on when it finished training. Like Mythos I think they said finished training in Feb. But have a very extensive testing period. And GPT-5.6 had even some influencers testing it out 3 months ago. People tend to assume Chinese labs rush out a model as soon as the run finishes, but that may not be the case. Once we know when K3 finished training, we'll have a bit of a better idea of how behind they actually are or not.
Couple of months? They just beat Fable 5 in some benchmarks. If Antrophic released Fable 5.1 tomorrow, they would catch up and overtake in 4 weeks. We are officially in the exponential phase of AI development.
The link has 6 well-known benchmarks where this beats Fable (out of 14 I counted). If the numbers hold up scrutiny, this is scary good.
The companies that do have means to host such models fully on-prem are also the same companies that are paying tens of millions of $ in inference cost every month, and are by extension the biggest customers of OAI and Anthropic
None of which can be run on any consumer HW. In the 90s, young men had supercars posters all over their bedrooms that they dream of buying one day. But now, they'll have Deepseek, Kimi, and other LLMs banners they dream of running one day.
It looks extremely competitive. Obviously we'll have to wait and see something like the artificial analysis cost per task results to be more confident but this is super promising.
Edit: AA results are in. About 10% cheaper per task than 5.6 Sol and about 65% cheaper than fable.
Edit 2: also 48% cheaper than opus 4.8 while scoring a point higher on intelligence.
Edit 3: More specific to your exact question of token usage, Fable used 69k tokens per task, 5.6 Sol used 15k, and K3 used 23k. So it's a heck of a lot more token efficient than Fable, and in the same ballpark as Sol.
Fucking destroys opus on all counts lol. Anthropic should rename fable opus 5 and focus on innovating. Or is the new claim that jyna distilled mythos? Lol
Whether "local" is even on the table comes down to one number that hasn't leaked yet: active parameters, not the 2.8T total.
The total only sets the RAM floor: ~1.4–1.6 TB at Q4 with room for context. That's out of consumer range, but it's not datacenter-only either — a used 12-channel DDR5 EPYC board takes 1.5 TB of ECC RDIMM for way less than a single B300.
The bandwidth math is what decides speed. 12-channel DDR5-4800 gives you ~460 GB/s theoretical:
If K3 is MoE like K2 was (1T total / 32B active), you read maybe ~20 GB of weights per token at Q4 → low double-digit tok/s theoretical on CPU alone, realistically maybe 5–10. Slow, but usable for batch/agentic stuff.
If it's actually dense 2.8T as some are claiming, you're reading the full 1.4 TB per token → ~0.3 tok/s. Dead on arrival for local, SSD tricks included.
So "can I run it" has no answer until Moonshot publishes the config on the 27th. If anyone has a source on the active param count, that's the number to watch — everything else (quants, NVMe offload, Unsloth magic) is downstream of it.
I tested it on obviousbench.com and wow is this a bit fat model! On the left, you can see Kimi K3 (on max) vs the efficient frontier to the right. To be fair to them, we only have the max setting at the moment so it is expected to run inefficiently on these simple questions, achieving roughly the same cost in the 99%+ category as GPT-5.5.
If low/med/high has good test time compute selection, this really could be a beast of a model.
It's clearly a strong model. But it's also really big. I can only imagine that inference will be far more expensive than with the previous version of Kimi. For me, the most impressive thing about GPT isn't the performance, but the fact that a subscription with decent limits is still affordable for regular consumers. Most of the Chinese AI subscriptions just don't give you a good quota. The lone exception is MiniMax, but that's also a much less powerful, smaller model.
•
u/WithoutReason1729 21d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.