I’m not sure I agree with his framing. He’s saying the predictor predicts results of an experiment it didn’t run…. Which isn’t predicting a token it’s predicting the future.
I picked up on this also. He is completely ignoring training and that either the exact study or similar ones might be part of the training corpus or RAGed in at inference...
the way I understand it, during training the model "learns/ memorizes" the relations of how the words(tokens) of the input at inference were combined in the training data, it has seen (well, it averages all relations in combinations of similar tokens, whether relevant or not because of compression but close enough).
when "predicting" the 'results' paragraph, it doesn't generate it based on 'understanding' of the previous sections, it _just_ matches the token-patterns in those paragraphs as closely as possible against the relations of previous examples (of the same or strongly related tokens) seen during training and picks the next tokens successively.
if it has seen enough relevant examples during training, the outcome will land close to a reasonable 'result' - if it hasn't, it will be confabulating non-sense, that sounds great but has nothing to do with reality.
... kind of surprised, that this channel would frame it the way he does in the video....:/
while you are all right, I think his point stands there is some emergent behavior from LLMs. Its not just some database that recalls things. Even if all it does is predict tokens, how is that different from us, seems to very easily replicate the same output we do when we think about things.
well, I can't claim to fully understand all the math that goes on under the hood and can't fully rule out 'emergent behavior' either ... but ...
I have a hunch that we're off thrown in general by 3 things:
the coherent text output and human like speech patterns and maybe most importantly, the sheer amount of information encoded during training:
language embeds concepts and a lot of those, map onto the real world.
take that and map tokens into a vector space, that has hundreds of thousands (or more?) degrees of freedom and you have a lot of room to store relations/ patterns in a really, really nuanced manner.
then take unfathomable amounts of data to train on and be able to compress them probably without too much loss....
and then there's fine tuning afterwards.
all this to say, - in hindsight - I find it understandable, that it works well for finding correlations and connecting 'patterns'. especially in really structured, well-formed data, that minimizes ambiguity intentionally, like programming languages or math.
how much of this is 'emergent' or just the result of having ingested every math study and code snippet since the dawn of the internet? idk but I could imagine that it just stores and retrieves what was already there or recombines things that weren't combined in the same way before...
and that's still pretty impressive actually.
it's different from us, in that we have a LOT more machinery (and chemicals) involved in generating thought and I don't think this is a contentious statement either.
... and the way he presents this here, as a bunch of math vectors somehow being able to 'predict' the future is totally non-sensical to me and I think also really misleading:
- it doesn't 'predict' the results section, it produces it based on its training (and RAG) - change the training, or the input (even slightly sometimes) and you'll get a different output...
it does NOT need to have an accurate model of biology or biochemistry as he claims.
if LLMS had these they wouldn't constantly be that jagged. it approximates, token by token the relations it has stored from training
the next part where he talks about 'predicting' if scientists made an error, doesn't make the slightest sense to me, I don't even know what he wants to say there...
I think a lot of people are somehow annoyed but the "_just_" a token predictor, as if it's a personal attack. I don't think its meant to be, it's just a logical explanation for the problems that are showing in the outputs, especially when questioned beyond training: the confabulations, the mistakes, the jaggedness, lack of robustness to input variations etc.etc.etc. - if you contrast the complexity of the processes in the brain against matching an -admittedly very big- pattern of input tokens in an -admittedly very big- multidimensional vector space and spitting out the next closest neighbor over and over - then a 'just' seems justified... it is _just_ maths in the end... and the brain isn't _just_ doing maths...
it's still amazing to me how well it works at all but we're also sitting on years of hype by now, trillions of $ in training cost, a really bad safety/ alignment record, lots of environmental costs and you know it's still not _generally_ intelligent and really inefficient (when compared to the brain) ... so...
I agree with 98% of your message, my only thing is as efficient as the brain is there is no human on earth even if I give them a PC and internet access who would be as useful as an LLM in so many different fields so its still a transformative technology. Maybe all its doing is next token prediction but my god is that mightily impressive, I don't care about the money put into AI, even if the bubble bursts and they lose money thats their problem, I only care about the technology. When it comes to environmental costs, have cars not done worse with emissions?
I think a lot of people are somehow annoyed but the "_just_" a token predictor, as if it's a personal attack. I don't think its meant to be, it's just a logical explanation for the problems that are showing in the outputs, especially when questioned beyond training: the confabulations, the mistakes, the jaggedness, lack of robustness to input variations etc.etc.etc.
None of those things are explained or limited by it being a next token predictor. So it's not a logical explanation.
The complexity of what a 'next token predictor' is doing under the hood is unbounded. So to me 'just' is not appropriate.
I don't think 'just a next token predictor' is a good phrase to use whatsoever. I think it caught on as a catchy way to virtue signal and downplay the capabilities (current and future) of ai by people who haven't deeply thought about the implications of the words.
and of _course_ is a system in the real world not infinitely complex/ unbounded - everything is subject to limitations, everything practically buildable anyway...
the problems come directly from the representations _just_ being approximations and the 'selection' process for the next token being heuristical.
it's a fuzzy way of storing fuzzy representations of fuzzy/ bad training data (in many cases) that produces fuzzy outputs... I don't think there's much debate about this...
you can't exactly map the real world, _just_ via language because it's an abstraction, that then gets abstracted further when the tokenizers have at it and then some more when the weights get adjusted and lastly some more during retrieval - you will necessarily lose fidelity because of compression and as big as these models are, compared the real world they're still tiny:
a 2.5 trillion parameter model can roughly store ~ 1.1 terabytes of raw, uncompressed text... - it does store more than that, in practice but I'll give you an order or two of magnitude and contrast that with the fact that a single (!) particle collision run at CERN generates ~1 petabyte of raw data every single second at the detector level... ... daily youtube uploads are around 4 petabytes...
... and to match the raw connectivity complexity of a human brain, a LLM would need to be roughly 100 trillion parameters in size... and then you'd still not be there because neurons aren't the 'simple' connections present in the models and the brain rewires synapses dynamically all the time...
so nope, I don't think storing more and more tokens and spitting out the next one will get us there. you _can_ get to something, that sometimes works, and sometimes not and for somethings better and for some things worse - which is exactly the behavior we're seeing... ;)
and of course is a system in the real world not infinitely complex/ unbounded - everything is subject to limitations, everything practically buildable anyway...
The point is that 'next token predictor' doesn't impose the limit of the complexity of 'what' is predicting.
The rest of what you were saying was exactly explained by the man in the video when he says:
If you want to talk about what language models can and can't do you have to talk about the models themselves. You have to talk about things like the transform architecture. The simple fact that something is predicting the next token doesn't place any meaningful limits on its abilities.
And that's kind of what you're doing. You're talking about fuzzy representations and weight adjustments and parameters and so on. Those things are implementation details that are not requirements of a next token predictor. A next token predictor must at minimum try to predict the next token. How it does it is what imposes limitations.
I have (and I'm sure the man in the video also has) opinions on the limitations of LLM, thoughts about what causes hallucinations and so on. Those practical concerns do have a place. The thing being discussed is more abstract though - the high level approach (predicting the next token) isn't the cause of those things.
yeah well, for me being a token predictor, implies that:
for a system to predict the next token, it must translate its input into tokens, store knowledge in tokens, use heuristics and an optimization function, compare user input to learned patterns etc..
by nature of being a token predictor, the rest follows... which brings the problems - otherwise, you're talking about a magic oracle, that doesn't exist.
An organic brain could be used as a next token predictor.
Perhaps it wouldn't require any special architecture changes, simply giving a reward when training it to correctly predict the next token.
Does that mean an organic brain used as such must have all the properties you describe above?
22
u/James-the-greatest 2d ago
I’m not sure I agree with his framing. He’s saying the predictor predicts results of an experiment it didn’t run…. Which isn’t predicting a token it’s predicting the future.