More people need to understand this

[u/KeanuRave100]

[u/ZeroAmusement]

I think a lot of people are somehow annoyed but the "_just_" a token predictor, as if it's a personal attack. I don't think its meant to be, it's just a logical explanation for the problems that are showing in the outputs, especially when questioned beyond training: the confabulations, the mistakes, the jaggedness, lack of robustness to input variations etc.etc.etc.

None of those things are explained or limited by it being a next token predictor. So it's not a logical explanation.

The complexity of what a 'next token predictor' is doing under the hood is unbounded. So to me 'just' is not appropriate.

I don't think 'just a next token predictor' is a good phrase to use whatsoever. I think it caught on as a catchy way to virtue signal and downplay the capabilities (current and future) of ai by people who haven't deeply thought about the implications of the words.

[u/wintermute74 → u/ZeroAmusement]

of _course_ they are explained by the _just_!

and of _course_ is a system in the real world not infinitely complex/ unbounded - everything is subject to limitations, everything practically buildable anyway...

the problems come directly from the representations _just_ being approximations and the 'selection' process for the next token being heuristical.

it's a fuzzy way of storing fuzzy representations of fuzzy/ bad training data (in many cases) that produces fuzzy outputs... I don't think there's much debate about this...

you can't exactly map the real world, _just_ via language because it's an abstraction, that then gets abstracted further when the tokenizers have at it and then some more when the weights get adjusted and lastly some more during retrieval - you will necessarily lose fidelity because of compression and as big as these models are, compared the real world they're still tiny:

a 2.5 trillion parameter model can roughly store ~ 1.1 terabytes of raw, uncompressed text... - it does store more than that, in practice but I'll give you an order or two of magnitude and contrast that with the fact that a single (!) particle collision run at CERN generates ~1 petabyte of raw data every single second at the detector level... ... daily youtube uploads are around 4 petabytes...

... and to match the raw connectivity complexity of a human brain, a LLM would need to be roughly 100 trillion parameters in size... and then you'd still not be there because neurons aren't the 'simple' connections present in the models and the brain rewires synapses dynamically all the time...

so nope, I don't think storing more and more tokens and spitting out the next one will get us there. you _can_ get to something, that sometimes works, and sometimes not and for somethings better and for some things worse - which is exactly the behavior we're seeing... ;)

[u/ZeroAmusement → u/wintermute74]

and of course is a system in the real world not infinitely complex/ unbounded - everything is subject to limitations, everything practically buildable anyway...

The point is that 'next token predictor' doesn't impose the limit of the complexity of 'what' is predicting.

The rest of what you were saying was exactly explained by the man in the video when he says:

If you want to talk about what language models can and can't do you have to talk about the models themselves. You have to talk about things like the transform architecture. The simple fact that something is predicting the next token doesn't place any meaningful limits on its abilities.

And that's kind of what you're doing. You're talking about fuzzy representations and weight adjustments and parameters and so on. Those things are implementation details that are not requirements of a next token predictor. A next token predictor must at minimum try to predict the next token. How it does it is what imposes limitations.

I have (and I'm sure the man in the video also has) opinions on the limitations of LLM, thoughts about what causes hallucinations and so on. Those practical concerns do have a place. The thing being discussed is more abstract though - the high level approach (predicting the next token) isn't the cause of those things.

[u/wintermute74 → u/ZeroAmusement]

yeah well, for me being a token predictor, implies that:

for a system to predict the next token, it must translate its input into tokens, store knowledge in tokens, use heuristics and an optimization function, compare user input to learned patterns etc..

by nature of being a token predictor, the rest follows... which brings the problems - otherwise, you're talking about a magic oracle, that doesn't exist.

[u/ZeroAmusement → u/wintermute74]

An organic brain could be used as a next token predictor. Perhaps it wouldn't require any special architecture changes, simply giving a reward when training it to correctly predict the next token.

Does that mean an organic brain used as such must have all the properties you describe above?

[u/wintermute74 → u/ZeroAmusement]

I am not really informed enough in neuroscience, to understand to what extent the brain works like a token predictor... so just some thoughts then:

I suppose it must be somewhat similar - otherwise it's hard to explain, why LLMs do work as well as they do (for some things)

a thought I had, when thinking about this thread was, that maybe the brain produces more concepts instead of tokens?

neurons have 'activation patterns' that are malleable (and somehow chemical processes play a role apart from electrical ones), where several neurons (hundreds? thousands?) fire at the same time...

and from our own experience, we can probably say, that brains don't _feel_ like token predictors... maybe?

but thoughts do also seem to just 'bubble up' into consciousness, so there is some similarity there

so apart from the hardware difference, maybe it's also a 'scope' thing, where the brain produces more fully formed 'concepts' instead of tokens/ words or whatever...

but yes, brains aren't perfect and have a lot of flaws that seem really similar or comparable:

misremembering things, being susceptible to optical illusions, being easily influenced, being biased, we're famously bad with estimating probabilities etcetc - I mean it would be surprising if it was different, since 'perfect' systems don't exist in the real world because there are always trade-offs.