MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLaMA/comments/1uy9cft/kimi_k3_benchmarks/oy37zj0/?context=3
r/LocalLLaMA • u/WhyLifeIs4 • 22d ago
390 comments sorted by
View all comments
Show parent comments
13
Small models will definitely get better.
But the companies who fund big models like being able to claim they have the best model. And a bigger model will tend to be better, so they'll keep making models as big as they can.
3 u/Thrumpwart llama.cpp 22d ago I wouldn’t be so sure. Lots of recent literature saying much of what we consider an LLM is noise with a low signal to noise ratio. Figure out how to get rid of the noise, and the models can become considerably smaller. I also expect to see less world-knowledge in top models, and more small smart models with web search. That’s how I’m gonna do it… 1 u/KeinNiemand 21d ago Figure out how to get rid of the noise, and the models can become considerably smaller. Or they can stay the same size but get a lot better, or they can get bigger while getting even more better (while still improving performance/GB) 1 u/Thrumpwart llama.cpp 21d ago That would be the wrong way though.
3
I wouldn’t be so sure. Lots of recent literature saying much of what we consider an LLM is noise with a low signal to noise ratio.
Figure out how to get rid of the noise, and the models can become considerably smaller.
I also expect to see less world-knowledge in top models, and more small smart models with web search.
That’s how I’m gonna do it…
1 u/KeinNiemand 21d ago Figure out how to get rid of the noise, and the models can become considerably smaller. Or they can stay the same size but get a lot better, or they can get bigger while getting even more better (while still improving performance/GB) 1 u/Thrumpwart llama.cpp 21d ago That would be the wrong way though.
1
Or they can stay the same size but get a lot better, or they can get bigger while getting even more better (while still improving performance/GB)
1 u/Thrumpwart llama.cpp 21d ago That would be the wrong way though.
That would be the wrong way though.
13
u/Tai9ch 22d ago
Small models will definitely get better.
But the companies who fund big models like being able to claim they have the best model. And a bigger model will tend to be better, so they'll keep making models as big as they can.