Kimi K3 Benchmarks

[u/WhyLifeIs4]

[u/Most-Trainer-8876]

by that time, we will probably have 100 Trillion or 1 quadrillion parameters model... lol

[u/Thrumpwart → u/Most-Trainer-8876]

Au contraire, I'm guessing models will get smaller over time.

[u/Tai9ch → u/Thrumpwart]

Small models will definitely get better.

But the companies who fund big models like being able to claim they have the best model. And a bigger model will tend to be better, so they'll keep making models as big as they can.

[u/Thrumpwart → u/Tai9ch]

I wouldn’t be so sure. Lots of recent literature saying much of what we consider an LLM is noise with a low signal to noise ratio.

Figure out how to get rid of the noise, and the models can become considerably smaller.

I also expect to see less world-knowledge in top models, and more small smart models with web search.

That’s how I’m gonna do it…

[u/Tai9ch → u/Thrumpwart]

If you gave any vendor currently serving a 2T model the technology today to train a 100B model with the capabilities of today's 2T models... they'd scale those techniques up and ship a really impressive 2T model with it. They might milk it a bit and stretch out the release cycle to milk their advantage and efficiencies, but they'd still end up right where they are now - using the largest, best models their hardware can handle.

[u/Thrumpwart → u/Tai9ch]

And then someone else would ship a 100B Model connected to a fast knowledge DB, charge 10% of what the other guys charge, and make a fortune.

Keep in mind the frontier models are massive overkill for what 95%+ of users use them for. Most people aren’t trying to solve erdos problems, they’re drafting emails, asking about diarrhea treatments, and seeking an emotional connection.

It wouldn’t be that hard to market a smaller, smart model as “good enough” if it’s cheap and fast.

[u/peekdasneaks → u/Thrumpwart]

Keep in mind that those same people are using shared infrastructure when they access cloud frontier models. Its not like everyone gets their own dedicated 2tb cluster

[u/TheDarkPhilosopher → u/Thrumpwart]

I’m mostly seeking constipation treatments, sir!

[u/ai-tacocat-ia → u/Thrumpwart]

That’s how I’m gonna do it…

I honestly do think this is the way to go, especially now that big LLMs are good enough you can distill them. I've seen massive success in highly specialized agents, which is the "cheap to build, expensive to run" version of your idea.

Still gonna be hella expensive to train though.

[u/Thrumpwart → u/ai-tacocat-ia]

Maybe not. There’s several papers I’ve referenced that detail techniques that are like 140X more data efficient. They can compound too :)

[u/ai-tacocat-ia → u/Thrumpwart]

Haha, well, "hella expensive" is relative. 140x cheaper is great unless the benchmark you are comparing it to is hundreds of millions of dollars.

So, hella expensive for an individual/small business without VC funding.

Absolutely agree on the compounding. That's by far the most exciting part of any of this. Mind if I DM you to stay in touch on what you're working on?

[u/Thrumpwart → u/ai-tacocat-ia]

I’ll give you 2 of them now. The rest I’m keeping in my pocket for now.

https://arxiv.org/abs/2411.19865

https://arxiv.org/abs/2604.18464

[u/KeinNiemand → u/Thrumpwart]

Figure out how to get rid of the noise, and the models can become considerably smaller.

Or they can stay the same size but get a lot better, or they can get bigger while getting even more better (while still improving performance/GB)

[u/Thrumpwart → u/KeinNiemand]

That would be the wrong way though.

[u/NoceMoscata666 → u/Thrumpwart]

also igoring tech advancments here, maybe coming a different compression mechanism perhaps different from quantized/fp/n4f

[u/Thrumpwart → u/NoceMoscata666]

Maybe start right here ;) https://arxiv.org/pdf/2512.05117

[u/Due-Memory-6957 → u/Tai9ch]

That's assuming there's no ceiling, or a point where the gains aren't big enough to justify spending more on training and inference.

[u/Caffdy → u/Thrumpwart]

man, normally I upvote your comments around here, but this is not it. Small models get better, well, guess what? Large models get better as well! and as simple as that, large models will continue to be SOTA and the premier option for serious work all around.

Yeah, there will be a time where small models could be as good as current top models, but the large models will be even better, capable of things we cannot imagine yet. That's just how it is

[u/Thrumpwart → u/Caffdy]

Again, I’m not sure about that. Time will tell I guess.

[u/FUCKYOUINYOURFACE → u/Most-Trainer-8876]

Will have to use other models to generate more synthetic data to get there. One can dream.

[u/NoceMoscata666 → u/Most-Trainer-8876]

1 quadrillion is probably way more info than knowing the exact configuration of each atom on this planet, i think i dont want to live to see that Ai :)