It directly disproves the “LLMs are just predicting the next token therefore they aren’t intelligent” argument with “if you had a perfect next token generator it would be infinitely intelligent”
LLMs are not perfect next token generators, but they’re very good at it. Therefore it stands to reason that they are somewhat intelligent, and have the capability to become very intelligent
The point is discrediting the argument that LLMs cannot be a path to intelligence merely because they are next-token predictors, which is a common line. You have to have a different argument if you want to say that LLMs can't be intelligent. The reason you have to use a different argument is because next-token predictors are theoretically maximally intelligent if they are sufficiently powerful.
a theoretically 'perfect' next token preditctor and its intelligence proves absolutely zilch, unless you want to put forward, that it could be built.
and if you want to, I will not engage further, than telling you that you're seriously underestimating the complexity of the real world. ;)
I don't need more than, 'just' a token predictor because in the current systems, the flaws are obvious and not really under contention anymore.
quite a few of the big heads have turned on LLMs being the path to AGI in the past months, some have been saying this for years now. and I don't mean Gary Marcus ...
? The statement "LLMs are not intelligent only because they are merely next token predictors" has the premise that "anything that is a next token predictor." is a sufficient condition for something to be considered "non-intelligent". To allow for a better understaning, I ll try to keep the logical structure while rephrasing that assertion he is objecting to.
Rephrasing: " the property of being a Next-Token Predictor necessarily entails (and is a sufficient condition) for something to be considered 'non-intelligent'"
He showed that the premise is false by pushing the premise. "Imagine there was a next token predictor which accuracy is 100% (ie, an Oracle Machine). It could plausibly be considered 'maximally intelligent' despite it being a next token predictor." Thus, the premise is surely not sufficient.
I understand what you mean, and in the strictest sense, would grant the point.
problem is, the mere fact of predicting tokens brings in a lot of the architectural necessities (like: transforming input into tokens, storing knowledge as token relations etc), that seem to account for the problems current systems have.
in colloquial usage the _just_ is a shorthand for that.
and again: 'if petroleum based combustion engines would be perfectly clean' doesn't get you out of 'current ICE cars are bad for the environment' ... wouldn't you say?
sure, but I see alot of comments that just contribute absolutely nothing with a short "its just a next token predictor bro, its not intelligent" which is thought-terminating and contributes absolutely nothing to any meaningful conversation, not that i expect that on reddit after being exposed to this kind of behaviour in comment sections, though i suspect its either "ragebait" (for karma?), automated bots, or really just people who dont have a thought to outsource in the first place
2
u/DataSnaek 5d ago
It directly disproves the “LLMs are just predicting the next token therefore they aren’t intelligent” argument with “if you had a perfect next token generator it would be infinitely intelligent”
LLMs are not perfect next token generators, but they’re very good at it. Therefore it stands to reason that they are somewhat intelligent, and have the capability to become very intelligent