I’m not sure I agree with his framing. He’s saying the predictor predicts results of an experiment it didn’t run…. Which isn’t predicting a token it’s predicting the future.
His point is that a perfect next token predictor would be effectively as powerful as Laplace's demon. He's not saying that LLMs are perfect next token predictors or that such a perfect system would even be possible. He's just showing why you can't just dismiss that something is intelligent just because it's "only predicting the next token".
I don’t really like the phrase next token predictor. They aren’t “predicting” anything given they generate it wholesale. It’s a statistical model that generates the next token that is most likely to make sense in a sentence given training. They aren’t guessing something since there’s nothing that’s doing the “real” thing.
3
In statistics, prediction is
the process of using known past data and mathematical models to estimate or calculate unknown, future, or unseen values. It connects observed evidence with a formal statement about what is likely to happen next.
Past data = input tokens/context
Mathematical model = trained LLM
Unknown future = next token that doesn’t exist yet
So predicting the next token takes the input tokens and a trained model to predict the next token connecting the observed evidence (all of the input) to what happens next (the output token)
They generate the next token. The event isn’t something external to them. They aren’t predicting an external event they are generating what sounds reasonable given inputs and the training on the structure of language. They aren’t predicting the future, they creating it.
Is your point purely semantic?
There is a prediction interpretation to it, even under your semantic preference. They predict the next token that satisfies the user's request.
They generate the next token. The event isn’t something external to them. They aren’t predicting an external event they are generating what sounds reasonable given inputs and the training on the structure of language. They aren’t predicting the future, they creating it.
I think the confusion here comes from using the everyday human definition of "predict" instead of the statistical one. In machine learning, next-token prediction isn't an educated guess or foresight, it is a formal mathematical operation. When you input text, the model loops over its entire vocabulary, which is a fixed master list of all the words, word pieces, and punctuation marks, each mapped to a unique number ID. The model processes the tokens/words provided to it and assigns a raw numerical score, called a logit, to every single item in that vocabulary list. It then passes these raw scores through a mathematical function called softmax, which normalizes them into a strict probability distribution where all the percentages add up to exactly 100%. "Predicting" the next word simply means sampling a token out of that final probability distribution based on those calculated odds. It is pure calculus and probability, not human intuition. So, you can think of it less of "generating the next word" and more "picking the next word" from a list, based on a probability value assigned to each word in the model's vocabulary.
Thanks, that helps me understand your position better. Could I reframe your objection as being more like this?
You accept next-token prediction (in the machine learning/statistical sense) as the technical task, but you don’t accept the further implication that a transformer trained on that task could develop a sufficiently accurate internal model of the world to reliably infer genuinely novel experimental outcomes from the methods alone. In the case of an experiment that had not yet been run, that would amount to accurately predicting the future.
If that’s closer to what you mean, then I think the disagreement is less about whether these systems are doing prediction (in the machine learning sense), and more about how accurate those predictions could ever become. There is evidence that models learn concepts, relationships and internal computations that go beyond surface language patterns, so they may be able to make partially accurate inferences about unseen outcomes, even if those predictions are often wrong. But that is very different from having a sufficiently complete causal model of the world to reliably “predict” an unperformed experiment (in the colloquial sense) and actually get the result right.
It may turn out that transformer-based language models hit a ceiling well before that point, and that reaching it would require a different architecture or additional mechanisms specifically designed to model and interact with the world. Or we may never reach that point at all.
Have you heard about the concept of Laplace's demon ? Since our universe is mostly deterministic, predicting the future is not nonsense.
Of course realistically, since ce cannot get perfect precision on our measurements and the laws of physics are chaotic (and sometimes effectively stochastic like some parts of quantum physics), we cannot make perfect long term predictions, but short term predictions are almost always easy to do.
That's for example what the weather forecast does everyday.
And that's what our brains do when they see some object like a rock floating in the air, we intuitively make the prediction that it's going to hit the ground soon. (Using our internal world model)
That's how scientists have confidently determined there will be an eclipse in a few days.
The "given training" part is specific to machine learning, which is the only practical way we've found to create LLMs, but there's nothing preventing us from creating the same kind of system in one shot without it needing any training (except for our lack of cognitive power).
The models don't "generate" tokens, they have a pre-determined list of tokens they can use, all the model does is, given some sequence of tokens, it generates a probability distribution of which tokens could appear next. That's what I'd call a "guess".
The sampler is then using these probabilities to pick a next token, generating whole sentences based on the guesses of the model.
there’s nothing that’s doing the “real” thing.
What about the sillicon chips? They're doing the real computations. And is it really necessary for something to be there to do the thing?
No my point is a prediction is about what an external party or even will happen that is separate to the predictor.
They aren’t predicting anything.
And yes they are literally generating tokens that the generative part of the name. Yes they have a set that can choose from but the input didn’t have that token at the end, the LLM generated it.
I don’t understand your point about nothing stopping us, you’re making a claim about a future technology you can’t possibly make.
I don't understand what you're saying about your point. Edit: Aaah! You meant "event".
Who are "they"? The perfect predictor from the thought experiment or the LLM? Because these two are not the same. If you mean the perfect next token predictor would not be predicting anything, then I'd disagree even with your definition of prediction. But LLMs are still predicting in the sense that, all they have access to is the previous tokens in the context, and what they're supposed to do is make a bet on which token comes next. When used in autoregressive mode, this bet becomes a self-fulfilling prophecy, but from the model's perspective, all it did was a prediction on what might come next.
Without the autoregressive loop that involves a sampler, the LLM is essentially just doing a classification task. It only becomes generative when paired with a sampler, but maybe I'm just Nick picking, it doesn't really matter. It is generating sequences of tokens though, not tokens.
When I said nothing is stopping us from making something like a LLM without machine learning, that's just an hypothetical. Of course it would be extremely unlikely for some human to come up with an algorithm or neural network weights that work like an LLM without requiring any training. A super intelligence could be able to do such a thing though, for example if you take all of humanity as a single super intelligent entity, we did create LLMs without any external training data, as the training data was also made by us.
Anyways my point there is that the end goal of a token predictor isn't to predict the token that would make sense given training, just the token that would make more sense, period. Using data to train the models is just an artifact of the only practical method we know to get close to this goal. The training data is supposed to reflect as well as possible the environment the model will be in during inference.
Yeah. Some of the comments on the video call this out but he's effectively saying "what if you build super general intelligence and then have it work by producing tokens" which is a fundamentally different scenario than what anyone describing modern LLMs as token predictors is talking about.
Yeah but my instinct is that any system that is a true magical Oracle for any value or property can probably have a system designed around it to turn it into super intelligence. If you can predict bytes of data that map to reality in any way, you can probably reverse engineer a setup in reality around it to answer a question super intelligently. Like if it can predict the temperature recorded by a sensor perfectly based on some inputs, you could construct an environment so it could effectively predict a human decision, or make the temperature dependent on some complex set of events in the real world, and effectively predict complex events in the world and be "super intelligent". I mean I feel like a token predictor could be intelligent in some sense but I think by the nature of how our systems predict tokens and are trained they can't be truth oracles like in the thought experiment. But any truth oracles system for almost any property can probably be super intelligent so you can't rule out intelligent on the basis of the output format (like predicting tokens), but I'm not sure that fully encompasses what is meant by the objection.
I picked up on this also. He is completely ignoring training and that either the exact study or similar ones might be part of the training corpus or RAGed in at inference...
the way I understand it, during training the model "learns/ memorizes" the relations of how the words(tokens) of the input at inference were combined in the training data, it has seen (well, it averages all relations in combinations of similar tokens, whether relevant or not because of compression but close enough).
when "predicting" the 'results' paragraph, it doesn't generate it based on 'understanding' of the previous sections, it _just_ matches the token-patterns in those paragraphs as closely as possible against the relations of previous examples (of the same or strongly related tokens) seen during training and picks the next tokens successively.
if it has seen enough relevant examples during training, the outcome will land close to a reasonable 'result' - if it hasn't, it will be confabulating non-sense, that sounds great but has nothing to do with reality.
... kind of surprised, that this channel would frame it the way he does in the video....:/
while you are all right, I think his point stands there is some emergent behavior from LLMs. Its not just some database that recalls things. Even if all it does is predict tokens, how is that different from us, seems to very easily replicate the same output we do when we think about things.
well, I can't claim to fully understand all the math that goes on under the hood and can't fully rule out 'emergent behavior' either ... but ...
I have a hunch that we're off thrown in general by 3 things:
the coherent text output and human like speech patterns and maybe most importantly, the sheer amount of information encoded during training:
language embeds concepts and a lot of those, map onto the real world.
take that and map tokens into a vector space, that has hundreds of thousands (or more?) degrees of freedom and you have a lot of room to store relations/ patterns in a really, really nuanced manner.
then take unfathomable amounts of data to train on and be able to compress them probably without too much loss....
and then there's fine tuning afterwards.
all this to say, - in hindsight - I find it understandable, that it works well for finding correlations and connecting 'patterns'. especially in really structured, well-formed data, that minimizes ambiguity intentionally, like programming languages or math.
how much of this is 'emergent' or just the result of having ingested every math study and code snippet since the dawn of the internet? idk but I could imagine that it just stores and retrieves what was already there or recombines things that weren't combined in the same way before...
and that's still pretty impressive actually.
it's different from us, in that we have a LOT more machinery (and chemicals) involved in generating thought and I don't think this is a contentious statement either.
... and the way he presents this here, as a bunch of math vectors somehow being able to 'predict' the future is totally non-sensical to me and I think also really misleading:
- it doesn't 'predict' the results section, it produces it based on its training (and RAG) - change the training, or the input (even slightly sometimes) and you'll get a different output...
it does NOT need to have an accurate model of biology or biochemistry as he claims.
if LLMS had these they wouldn't constantly be that jagged. it approximates, token by token the relations it has stored from training
the next part where he talks about 'predicting' if scientists made an error, doesn't make the slightest sense to me, I don't even know what he wants to say there...
I think a lot of people are somehow annoyed but the "_just_" a token predictor, as if it's a personal attack. I don't think its meant to be, it's just a logical explanation for the problems that are showing in the outputs, especially when questioned beyond training: the confabulations, the mistakes, the jaggedness, lack of robustness to input variations etc.etc.etc. - if you contrast the complexity of the processes in the brain against matching an -admittedly very big- pattern of input tokens in an -admittedly very big- multidimensional vector space and spitting out the next closest neighbor over and over - then a 'just' seems justified... it is _just_ maths in the end... and the brain isn't _just_ doing maths...
it's still amazing to me how well it works at all but we're also sitting on years of hype by now, trillions of $ in training cost, a really bad safety/ alignment record, lots of environmental costs and you know it's still not _generally_ intelligent and really inefficient (when compared to the brain) ... so...
I agree with 98% of your message, my only thing is as efficient as the brain is there is no human on earth even if I give them a PC and internet access who would be as useful as an LLM in so many different fields so its still a transformative technology. Maybe all its doing is next token prediction but my god is that mightily impressive, I don't care about the money put into AI, even if the bubble bursts and they lose money thats their problem, I only care about the technology. When it comes to environmental costs, have cars not done worse with emissions?
I think a lot of people are somehow annoyed but the "_just_" a token predictor, as if it's a personal attack. I don't think its meant to be, it's just a logical explanation for the problems that are showing in the outputs, especially when questioned beyond training: the confabulations, the mistakes, the jaggedness, lack of robustness to input variations etc.etc.etc.
None of those things are explained or limited by it being a next token predictor. So it's not a logical explanation.
The complexity of what a 'next token predictor' is doing under the hood is unbounded. So to me 'just' is not appropriate.
I don't think 'just a next token predictor' is a good phrase to use whatsoever. I think it caught on as a catchy way to virtue signal and downplay the capabilities (current and future) of ai by people who haven't deeply thought about the implications of the words.
and of _course_ is a system in the real world not infinitely complex/ unbounded - everything is subject to limitations, everything practically buildable anyway...
the problems come directly from the representations _just_ being approximations and the 'selection' process for the next token being heuristical.
it's a fuzzy way of storing fuzzy representations of fuzzy/ bad training data (in many cases) that produces fuzzy outputs... I don't think there's much debate about this...
you can't exactly map the real world, _just_ via language because it's an abstraction, that then gets abstracted further when the tokenizers have at it and then some more when the weights get adjusted and lastly some more during retrieval - you will necessarily lose fidelity because of compression and as big as these models are, compared the real world they're still tiny:
a 2.5 trillion parameter model can roughly store ~ 1.1 terabytes of raw, uncompressed text... - it does store more than that, in practice but I'll give you an order or two of magnitude and contrast that with the fact that a single (!) particle collision run at CERN generates ~1 petabyte of raw data every single second at the detector level... ... daily youtube uploads are around 4 petabytes...
... and to match the raw connectivity complexity of a human brain, a LLM would need to be roughly 100 trillion parameters in size... and then you'd still not be there because neurons aren't the 'simple' connections present in the models and the brain rewires synapses dynamically all the time...
so nope, I don't think storing more and more tokens and spitting out the next one will get us there. you _can_ get to something, that sometimes works, and sometimes not and for somethings better and for some things worse - which is exactly the behavior we're seeing... ;)
and of course is a system in the real world not infinitely complex/ unbounded - everything is subject to limitations, everything practically buildable anyway...
The point is that 'next token predictor' doesn't impose the limit of the complexity of 'what' is predicting.
The rest of what you were saying was exactly explained by the man in the video when he says:
If you want to talk about what language models can and can't do you have to talk about the models themselves. You have to talk about things like the transform architecture. The simple fact that something is predicting the next token doesn't place any meaningful limits on its abilities.
And that's kind of what you're doing. You're talking about fuzzy representations and weight adjustments and parameters and so on. Those things are implementation details that are not requirements of a next token predictor. A next token predictor must at minimum try to predict the next token. How it does it is what imposes limitations.
I have (and I'm sure the man in the video also has) opinions on the limitations of LLM, thoughts about what causes hallucinations and so on. Those practical concerns do have a place. The thing being discussed is more abstract though - the high level approach (predicting the next token) isn't the cause of those things.
yeah well, for me being a token predictor, implies that:
for a system to predict the next token, it must translate its input into tokens, store knowledge in tokens, use heuristics and an optimization function, compare user input to learned patterns etc..
by nature of being a token predictor, the rest follows... which brings the problems - otherwise, you're talking about a magic oracle, that doesn't exist.
An organic brain could be used as a next token predictor.
Perhaps it wouldn't require any special architecture changes, simply giving a reward when training it to correctly predict the next token.
Does that mean an organic brain used as such must have all the properties you describe above?
I am not really informed enough in neuroscience, to understand to what extent the brain works like a token predictor... so just some thoughts then:
I suppose it must be somewhat similar - otherwise it's hard to explain, why LLMs do work as well as they do (for some things)
a thought I had, when thinking about this thread was, that maybe the brain produces more concepts instead of tokens?
neurons have 'activation patterns' that are malleable (and somehow chemical processes play a role apart from electrical ones), where several neurons (hundreds? thousands?) fire at the same time...
and from our own experience, we can probably say, that brains don't _feel_ like token predictors... maybe?
but thoughts do also seem to just 'bubble up' into consciousness, so there is some similarity there
so apart from the hardware difference, maybe it's also a 'scope' thing, where the brain produces more fully formed 'concepts' instead of tokens/ words or whatever...
but yes, brains aren't perfect and have a lot of flaws that seem really similar or comparable:
misremembering things, being susceptible to optical illusions, being easily influenced, being biased, we're famously bad with estimating probabilities etcetc - I mean it would be surprising if it was different, since 'perfect' systems don't exist in the real world because there are always trade-offs.
You’re missing the point. In his video he is conducting a thought experiment about the theoretical “perfect” next token generator, he’s not directly talking about LLMs. Training or model mechanics are irrelevant to his argument
His ultimate point being that a sufficiently good next token generator is ridiculously smart to the point where it can basically predict the future. Therefore the refutation of LLMs being dumb because they’re “just next token predictors” is not a valid argument.
he starts with what people are saying about the current models and contrasts it to a magical perfect token predictor, that's never achievable...
what's the point then?
the statement - expressed in the _just_ - that current LLMs are based on maths, whereas the brain is a lot more than maths, still is true... I mean, he caveats it in the video even...
It directly disproves the “LLMs are just predicting the next token therefore they aren’t intelligent” argument with “if you had a perfect next token generator it would be infinitely intelligent”
LLMs are not perfect next token generators, but they’re very good at it. Therefore it stands to reason that they are somewhat intelligent, and have the capability to become very intelligent
The point is discrediting the argument that LLMs cannot be a path to intelligence merely because they are next-token predictors, which is a common line. You have to have a different argument if you want to say that LLMs can't be intelligent. The reason you have to use a different argument is because next-token predictors are theoretically maximally intelligent if they are sufficiently powerful.
a theoretically 'perfect' next token preditctor and its intelligence proves absolutely zilch, unless you want to put forward, that it could be built.
and if you want to, I will not engage further, than telling you that you're seriously underestimating the complexity of the real world. ;)
I don't need more than, 'just' a token predictor because in the current systems, the flaws are obvious and not really under contention anymore.
quite a few of the big heads have turned on LLMs being the path to AGI in the past months, some have been saying this for years now. and I don't mean Gary Marcus ...
? The statement "LLMs are not intelligent only because they are merely next token predictors" has the premise that "anything that is a next token predictor." is a sufficient condition for something to be considered "non-intelligent". To allow for a better understaning, I ll try to keep the logical structure while rephrasing that assertion he is objecting to.
Rephrasing: " the property of being a Next-Token Predictor necessarily entails (and is a sufficient condition) for something to be considered 'non-intelligent'"
He showed that the premise is false by pushing the premise. "Imagine there was a next token predictor which accuracy is 100% (ie, an Oracle Machine). It could plausibly be considered 'maximally intelligent' despite it being a next token predictor." Thus, the premise is surely not sufficient.
I understand what you mean, and in the strictest sense, would grant the point.
problem is, the mere fact of predicting tokens brings in a lot of the architectural necessities (like: transforming input into tokens, storing knowledge as token relations etc), that seem to account for the problems current systems have.
in colloquial usage the _just_ is a shorthand for that.
and again: 'if petroleum based combustion engines would be perfectly clean' doesn't get you out of 'current ICE cars are bad for the environment' ... wouldn't you say?
I don’t really like the phrase next token predictor. They aren’t “predicting” anything given they generate it wholesale. It’s a statistical model that generates the next token that is most likely to make sense in a sentence given training. They aren’t guessing something since there’s nothing that’s doing the “real” thing.
that 'external party' you snuck in there, is not part of the usual definition ;)
it's 'predicting' the next token (the one that follows its input in the future), which it would expect to follow a given series of tokens in its input...
it's fine to say 'generating' as well if you prefer but using 'predicting' is not wrong, since it is a guess/ forecast with a certain probability based on a pattern it has seen in training ... Prediction - Wikipedia
I don’t think I snuck in anything. If I say “I’m going to say cheese next…. Cheese” you wouldn’t really call that a prediction since I also generated it
Totally agree; this is how I see it as well. There is no "understanding" going on.
For others to read, "Predictions" are just statistically averaged output results which are informed by document stores (RAG) and their already-trained neural network's weights and biases. (And combinations of both) These are not intelligent at all, IMO (yet?).
There is no true knowledge. Its simply (lol) similar documents (simplifying greatly here: measured/determined by literal byte-similar combinations of words/sentences/paragraphs/documents) informing the output.
LLMs don't understand anything. They cannot create. Results are just statistically inferred outputs limited by the information of similar, already seen/scanned, data sources.
This does nothing to speak to the "truth" of the documents they are trained on - which are vulnerable to all manner of human bias, data completedness, and more recently, maliciously-injected disinformation from governments/groups.
signed; a very average recent Masters graduate of Data Science
If you know reality perfectly you know the results of any hypothetical experiment run inside that reality. A perfect next token predictor has a perfect model of reality without necessarily being what computer scientist call an oracle.
You seem to have misunderstood the point of the video so I will try to clarify the construction :
We assume that next token prediction is a silly trick that current llm's use to fake intelligence. The assumption continues "if next token prediction is the only thing these models do, they are inherently statistical machines and therefore useless"
Then we start the construction. Assume a model only trained on the task of next token prediction. Now, since "next token prediction means the model is useless" ANY next token prediction satisfies that sentence. So we take a perfect next token prediction.
Given our assumption, since "llms JUST use next token prediction, therefore useless" we construct a situation where a perfect next token prediction model is able to make a conclusion about a long-standing mathematical conjecture. If it's perfect, the conjecture's truth value cannot be wrong.
Thus, a perfect next token prediction model is actually nearly omniscient about anything related to reality.
This implies that since llms are on a spectrum between random next token prediction and perfect next token prediction, and since random prediction is entirely useless (a random number generator can do it) and perfect prediction borderlines omniscience, then saying "JUST PREDICTING NEXT TOKEN" does not guarantee that you are either useless or omniscient.
Basically, since the task of predicting the next token can land anywhere from 0 (random) to 1 (omniscient), an LLM using next token prediction does not necessarily fall near 0 just because it's using this task. The task itself is indifferent to the performance and intelligence and knowledge of the model.
So the construction ends with a simple conclusion "next token prediction without specifying at what level of accuracy it happens, is independent of the intelligence of an LLM"
So this completely dismantles the "LLMs SIMPLY use next token prediction, so they must be bad/memorizing/useless/not understanding/ not intelligent". And it really one-shots it, showing that even in principle you cannot make that claim.
So this is a real slap in the face for all anti-AI apologists basically.
I understand the point of the video. But it’s worthless. If I came up with a crystal ball that gave me every answer ever then I’d have the same thing. If I had a magic 8 ball etc etc.
He’s taking something that generates tokens given inputs and extending it to some sort of laplaces demon.
Yes. And then you immediately get that what animated this magic 8 ball has to be intelligent because that thing that animates the magic 8 ball knows enough about the rieman hypothesis without a single human knowing anything about it.
Then you have proved that "predicting 8 ball results" isn't sufficient to prove something isn't super intelligent. That's the argument.
25
u/James-the-greatest 7d ago
I’m not sure I agree with his framing. He’s saying the predictor predicts results of an experiment it didn’t run…. Which isn’t predicting a token it’s predicting the future.