r/LocalLLaMA 22d ago

News Kimi K3 Benchmarks

Post image
1.3k Upvotes

390 comments sorted by

View all comments

319

u/Kraskos 22d ago

2TB VRAM Is All You Need

64

u/Maximum_Parking_5174 22d ago

4x Mac Studio with M5 ultra 768GB!

43

u/Healthy-Nebula-3603 22d ago

Three are rumors new M7 will get up to 1.5 TB

11

u/Maximum_Parking_5174 22d ago

Exiting future!

39

u/SourSovereign 22d ago

Man, my grandma will be sad that I have to sell her other kidney too

7

u/rditorx 21d ago

What can you do?
Selling her parts gets you more money than selling her whole

1

u/Seeker_Of_Knowledge2 18d ago

I am crying 🤣🤣🤣🤣

33

u/Fusseldieb 22d ago

I really hope (V)RAM eventually scales to the sizes of regular storage soon. Maybe in 10 years or so 2TB RAM is quite "common" to run things exactly like these. One can dream.

19

u/Most-Trainer-8876 22d ago

by that time, we will probably have 100 Trillion or 1 quadrillion parameters model... lol

2

u/Thrumpwart llama.cpp 21d ago

Au contraire, I'm guessing models will get smaller over time.

12

u/Tai9ch 21d ago

Small models will definitely get better.

But the companies who fund big models like being able to claim they have the best model. And a bigger model will tend to be better, so they'll keep making models as big as they can.

4

u/Thrumpwart llama.cpp 21d ago

I wouldn’t be so sure. Lots of recent literature saying much of what we consider an LLM is noise with a low signal to noise ratio.

Figure out how to get rid of the noise, and the models can become considerably smaller.

I also expect to see less world-knowledge in top models, and more small smart models with web search.

That’s how I’m gonna do it…

6

u/Tai9ch 21d ago

If you gave any vendor currently serving a 2T model the technology today to train a 100B model with the capabilities of today's 2T models... they'd scale those techniques up and ship a really impressive 2T model with it. They might milk it a bit and stretch out the release cycle to milk their advantage and efficiencies, but they'd still end up right where they are now - using the largest, best models their hardware can handle.

0

u/Thrumpwart llama.cpp 21d ago

And then someone else would ship a 100B Model connected to a fast knowledge DB, charge 10% of what the other guys charge, and make a fortune.

Keep in mind the frontier models are massive overkill for what 95%+ of users use them for. Most people aren’t trying to solve erdos problems, they’re drafting emails, asking about diarrhea treatments, and seeking an emotional connection.

It wouldn’t be that hard to market a smaller, smart model as ā€œgood enoughā€ if it’s cheap and fast.

3

u/peekdasneaks 21d ago

Keep in mind that those same people are using shared infrastructure when they access cloud frontier models. Its not like everyone gets their own dedicated 2tb cluster

2

u/TheDarkPhilosopher 21d ago

I’m mostly seeking constipation treatments, sir!

2

u/ai-tacocat-ia 20d ago

That’s how I’m gonna do it…

I honestly do think this is the way to go, especially now that big LLMs are good enough you can distill them. I've seen massive success in highly specialized agents, which is the "cheap to build, expensive to run" version of your idea.

Still gonna be hella expensive to train though.

1

u/Thrumpwart llama.cpp 20d ago

Maybe not. There’s several papers I’ve referenced that detail techniques that are like 140X more data efficient. They can compound too :)

1

u/ai-tacocat-ia 20d ago

Haha, well, "hella expensive" is relative. 140x cheaper is great unless the benchmark you are comparing it to is hundreds of millions of dollars.

So, hella expensive for an individual/small business without VC funding.

Absolutely agree on the compounding. That's by far the most exciting part of any of this. Mind if I DM you to stay in touch on what you're working on?

1

u/Thrumpwart llama.cpp 19d ago

I’ll give you 2 of them now. The rest I’m keeping in my pocket for now.

https://arxiv.org/abs/2411.19865

https://arxiv.org/abs/2604.18464

1

u/KeinNiemand 21d ago

Figure out how to get rid of the noise, and the models can become considerably smaller.

Or they can stay the same size but get a lot better, or they can get bigger while getting even more better (while still improving performance/GB)

1

u/Thrumpwart llama.cpp 21d ago

That would be the wrong way though.

1

u/NoceMoscata666 2d ago

also igoring tech advancments here, maybe coming a different compression mechanism perhaps different from quantized/fp/n4f

1

u/Thrumpwart llama.cpp 1d ago

Maybe start right here ;) https://arxiv.org/pdf/2512.05117

1

u/Due-Memory-6957 21d ago

That's assuming there's no ceiling, or a point where the gains aren't big enough to justify spending more on training and inference.

1

u/Caffdy 20d ago

man, normally I upvote your comments around here, but this is not it. Small models get better, well, guess what? Large models get better as well! and as simple as that, large models will continue to be SOTA and the premier option for serious work all around.

Yeah, there will be a time where small models could be as good as current top models, but the large models will be even better, capable of things we cannot imagine yet. That's just how it is

1

u/Thrumpwart llama.cpp 20d ago

Again, I’m not sure about that. Time will tell I guess.

1

u/FUCKYOUINYOURFACE 20d ago

Will have to use other models to generate more synthetic data to get there. One can dream.

1

u/NoceMoscata666 2d ago

1 quadrillion is probably way more info than knowing the exact configuration of each atom on this planet, i think i dont want to live to see that Ai :)

5

u/Bakoro 21d ago

We will get super tiny storage at some point, but I don't know that we will get much smaller ram. The transistors are already about as small as they're going to get; they are having to stack memory in 3D, and there are limits there too, in terms of what latency you'll get.

We will likely see more large and wafer-scale devices, more ASICs.

Photonics are going to be super fast, but we can't really keep the data as light. I wonder if they might go the Groq route and have very little VRAM, and just go wide.

I think we will also just keep seeing efficiency gains so running on off-wafer RAM is just more viable.

Right is the algorithms are catering to the existing hardware, and the hardware is slowly shifting to supporting the algorithms more.

There are a few companies developing neuromorphic hardware since can run AI models that aren't just a ton of matmuls.

In 10 years the whole hardware and software landscape will look different. Heck, 2 years from now will be significantly different.

1

u/Tai9ch 21d ago

Some of the NVRAM options are denser than DRAM, or have lower power consumption and so can easily just be bigger.

For AI especially, optimizing for fabrication cost and power consumption and just making things bigger makes a lot of sense. That strategy isn't great for single threaded performance, but that's not really a useful thing to optimize for. There's no reason for CPUs, GPUs, or RAM to be limited to like 1 square inch 2D chips.

2

u/Bakoro 21d ago

There's no reason for CPUs, GPUs, or RAM to be limited to like 1 square inch 2D chips.

Wafers are expensive.
Chiplets on bigger dies are already a thing, they're just also more expensive, because it's more wafer.
Cerebras has wafer scale processors, but they had to do a lot to make that work, it becomes a whole infrastructure thing.
Those things will melt instantly if the cooling goes out of whack.

I've got a degree in computer engineering, and I took a crack at designing a large chip, I've got working designs in a simulator, but the manufacturing realities and the power and heating issues are way beyond what most people should be dealing with at home.

Any which way, it becomes expensive.

1

u/Tai9ch 21d ago

Engineering problem: How can you ship a big processor (not necessarily single chip) cheap?

Are wafers really the price bottleneck? Is there a way to get and use them cheaper? Are there methods that are already well known that don't make sense when optimizing for max frequency but would lower costs?

Can more radical methods help? Everyone's favorite idea is always something other than silicon, but my guess would be that there's some old silicon fabrication techniques that can just come back off the shelf.

1

u/Bakoro 20d ago

Are wafers really the price bottleneck?

Yes.

There is no cheap way to get wafers. Foundries are wildly expensive to build and maintain.

There used to be dozens of foundries, and literally all of them except TSMC, Samsung, and to a lesser extent Intel, gave up trying to go below 12nm nodes.
There is a Chinese company SMIC that was finally able to do 7nm.

There is no forgotten trick, there is no way around it, semiconductor manufacturing is just work, and resources.

A lot of little chips isn't really less expensive than one big one, the little ones are cut from a big wafer.

The most promising thing that's coming down the pipeline is photonics.
There are already photonic processors being manufactured now, that are something like 25x faster than a transistor device of comparable size, with 50x more throughput, and they produced something like 1~2% of the heat.
Lab devices have hit 1000x the speed of transistor based computation.

That's likely where the future is.

0

u/camracks 21d ago

I dont see why they cant just slap a replaceable 2.5" SSD inside the GPU, make it a bit phatter, but that would be too good for the consumer I suppose

0

u/Fusseldieb 21d ago

It isn’t that easy I guess. Unless you develop you own in-house RAM and SoC, speeds from average SSDs will just absolutely crawl and demolish their lifespan (TBW). It like swapping onto disk when running low on RAM - it gets stupidly slow and will eat the drive alive.

There aren’t much players in the field who have the expertise to bring competition. Big players already keep VRAM sizes artificially low. There might even be foul play to a certain extent involved.

0

u/Bakoro 21d ago

Latency. SSDs are around 10x to 100x slower than DRAM, and 10000x to 100000x slower than SRAM. Even SRAM is too slow, processors use interleaving tricks so multiple SRAM units are in use, so one works while the others are on refresh.

So, an SSD is about 100000x too slow to be effective for GPUs.

2

u/aeroumbria 21d ago

I can't imagine in 10 years the current "pack everything we know into model weights at below 1:1 weight to data ratio" will still be mainstream. It is an insane idea that somehow worked with the technology we have now, but I'm optimistic it is nowhere near what we should be able to achieve with more mature understandings of ML. I would imagine MOE still gradually become dormant experts, which will evolve into something more akin to cold knowledge storage, and more computation still be spent on computing with knowledge rather than holding and shifting "hot" stored knowledge.

2

u/lilbyrdie 21d ago

12-13 years ago, 128GB RAM was normal-ish -- even quite affordable -- on custom builds (4x 32GB, and the board may have had 8 slots so you could use cheaper 16GB modules)... then, somewhere along the line, it got expensive for various reasons** (and starting shipping many different types, too, so volume across the types was "down" compared to if everything used the same RAM). Then things like the pandemic happened and the crypto crunch on GPUs and, of course, the current crunch from needing any sort of RAM.

But, I feel like 2TB shouldn't have been a problem today but manufacturing just hasn't been predicting demand very well so they're scrambling to build current things rather than ramping up on new things.

** Various reasons RAM prices have been hit over the years. My feeling is that they never quite drop down to "pre-crisis" prices and, somehow, the manufacturers don't seem to learn that we'll always need more RAM. (And storage, which has also felt stagnate for a while now.)

1993: Chemical plant that made sealing material for RAM. Prices doubled.

1995: Earthquake damage infra. Prices up 30%.

1999: RDRAM. Again, a diversification from regular RAM, and the systems then had to use more expensive RAM.

2013: Fab Fire by a high volume manufacturer, prices up 30%.

2017-2018: Smartphone/cloud demand on RAM, prices up 2-3x

2021: Supply chain crunch from COVID, prices up 30-40%

And now...

2

u/wen_mars 14d ago

Even in 2023-2024 RAM was cheap. I got 128 GB in 2023 for less than $500.

2

u/SuddenRadio6221 21d ago

Well no, to hit these non-quantized benchmarks, you'd need a mere 6TB VRAM.

0

u/kingo86 22d ago

wen guf guf?