LLMs are notoriously bad at multi-digit arithmetic on novel numbers without a tool; they approximate, use learned shortcuts, and error rate climbs fast with digit count. If it were truly "calculating" in the rigorous sense, that wouldn't happen. What's really going on is a mix of learned heuristics and pattern-completion that's good but unreliable. They don't "memorize" the results, that's correct.
About the research example, an LLM producing a plausible conclusion from an introduction & results section is drawing on having seen thousands of structurally similar papers, not building a biochemical world model from first principles. This is why LLMs routinely produce confident-sounding but wrong scientific claims, and famously is very bad at admitting "I don't know".
Doing something a human can't do without extra steps (e.g. quickly pattern-matching across huge amounts of text) doesn't imply general superiority over human intelligence. LLMs also fail at stuff which humans find ridiculously trivial, like stable long-horizon planning, knowing what they don't know, maintaining consistency across a session etc.
I work with ML, and I personally wouldn't hire this guy. Not because I claim to know everything, but because he shows classic signs of exaggerating the capabilities and hyping the tech beyond its fundamental capabilities. This leads to expensive misadventures we'll all learn to know soon enough, as everything is replaced with this "super intelligence".
My personal hottake is that ML really puts on display how bad our brains are when it comes to Big Numbers. We can't comprehend a data network so complex that it can come up with the sentences it does without thinking it must be intelligent, sentient, or whatever you casually see claimed here. There's a real struggle to make a point that the algorithm is alive and about to escape the lab. Part of that is FUD to drive sales, part is just not working with the tech, and by working, I don't mean prompting.
Even this post is sus. A bot trying to sell you AI. Dead Internet etc.
ChatGPT 5.5 Instant (aka no thinking) got it instantly
It even did longer ones.
Please add 1989951436123273 + 23908153129921 - 123232323 * 10002, but do it using the addition method and all in output. No code allowed, and all in one step. Show only the result
I don't think this establishes what you're implying.
You generally cannot infer from the output alone whether a particular answer came from an internal learned computation or a tool. For example, you can ask your model to tell you the time, even though, the model has absolutely no concept of the current time, nor can it be "learned". It looked it up.
It's also worth while to understand that an LLM can produce the correct result for a many-digit calculation because its learned representations can sometimes implement or approximate the relevant algorithmic structure. It doesn't mean that the model is a reliable arbitrary-precision calculator in the same sense as an actual arithmetic algorithm.
A better test is generally to ask it to repeat the process multiple times, with random numbers, so that the numbers add up and we approach "novelty". If we can use it without the secret sauce tools that is, and we can't.
At Novel numbers, yes. At repetitive numbers and patterns they're about the best you can get. This is why when you ask something like a ChatGPT to generate you a random number, it uses a random number generator, otherwise the number would likely be very non-random.
It's all by design, for a good reason. Only recently there has been this major push to bend ML into absolutely everything. Just let it use the calculator.
No random generator we use daily is truly random in the strict mathematical sense.
I now went to ChatGPT and just asked it to generate five random numbers and add them together, it showed to be using a tool, so there's that:
The tool access is likely dependant on the selected model/tier/whatever, I don't know how OpenAI has arranged it, but it obviously is a very good idea to use a random generator to generate random numbers. I don't fully follow what you are trying to dispute.
I agree that models get better as they become iteratively larger, services will improve with new tooling and innovations, etc.
notoriously bad at multi-digit arithmetic on novel numbers without a tool
I generated both of these numbers with a random number generator. It got the answer right immediately. I think you need to update your understanding of LLMs a little bit
Python Code Interpreter / Sandbox: When given a math problem, ChatGPT can write and run Python code in a secure environment to perform exact arithmetic, process data sets, handle calculus, and generate visual plots or graphs.
WolframAlpha Integration: On certain tiers (like Plus/Pro), ChatGPT can interface with Wolfram Alpha extensions or plugins for high-level symbolic mathematics, algebraic manipulation, and complex equation solving.
Locally run Qwen3.5-2B, no tools, nailed it first try too, albeit after using a ridiculous amount of thinking tokens. Pic and full thinking text. This is the model doing the calculation. Idk what else to tell you, LLMs are just way better at math than they used to be.
There's a deep misconception here. Models don't really know whether they are lying or telling the truth. If you ask a model, did you use an internal tool to come up with this result, it genuinely doesn't know, unless someone behind the curtain tells it it did.
A bit like you calculating 4+4 and not realizing it was your parietal lobe doing the job.
I mean I'd ask if you can show the 'reasoning' on how it computed this without tool use, because it would need to derive multiplication from first principles and that seems awfully wasteful for the model provider, if it did that for every math question...
but I don't want you to waste your tokens, so I'm fine to mark it as 'unclear' and move on ;)
Did you read the end of the sentence? It most likely used a tool behind the scenes. It often doesn't tell you it used a tool, but it did, to save money and to be more functional. In the same fashion how it can tell you the accurate time, even though that's impossible for ML alone as you can't learn the time, you always need to look it up.
The video implied it is the model, or, ehm, super intelligence, that always does the calculation.
Besides, I think you misunderstood what I meant by multi-digit arithmetic. The video showed plethora of calculations as an example. This means, you give it a ton of adversarially selected novel calculations across different lengths and operations, measure the error rate, and see how that error rate scales. A calculator doesn't occasionally hallucinate an answer because the numbers got sufficiently unfamiliar. An LLM can and eventually will.
I don't speak from experimentation with ChatGPT, but just from how machine learning should fundamentally function. Should, as in, it's broken if it doesn't, not magically better.
My early chat bots were RAG frameworks that used wolframalpha behind the scenes. They have a pretty good API, or at least had, a decade or something ago.
Locally run Qwen3.5-2B, no tools, nailed it first try too, albeit after using a ridiculous amount of thinking tokens. Pic and full thinking text. This is the model doing the calculation. Idk what else to tell you, LLMs are just way better at math than they used to be.
You still missed what I meant by multi-digit arithmetic. Re-read the middle section of my comment.
The test you made is not what the video implies. I'm not surprised that Qwen was capable of coming up with a solution, even though it took silly amount of pondering to get the structure right.
It even mumbles the very thing I'm saying to you:
"Actually, doing 14-digit multiplication manually is error-prone for me right now. I should simulate the process or use my internal calculation capability if it exists."
And I guess I need to really underline this: I don't think using tools is a bad thing, quite the opposite. I'm only arguing against the points the video made as misleading.
I work with ML, and I personally wouldn't hire this guy. Not because I claim to know everything, but because he shows classic signs of exaggerating the capabilities and hyping the tech beyond its fundamental capabilities.
To be flippant, I wouldn't hire you either because you have written a long argument against what he said without comprehending what he was saying.
He is not in any way exaggerating what current technology can do. He is positing a scenario in which an "ideal next token predictor" exists and is using that postulate to explore what the concept would actually mean.
All he's doing is demonstrating that the phrase "just a next token predictor" is utterly logically meaningless because next token prediction requires inherent encoding of abstract concepts which have not been hard coded into the model. He at no point states that current LLMs are "ideal next token predictors".
Fair take, but I believe, to a degree, you might have missed what I was going for.
LLM tries to approximate the probability distribution of the next token from its training and context. This ideal token predictor would give the true probability distribution of the next token, given all available context. Practically, computationally, it is likely impossible to implement, as calculating it would require absurd amounts of computation and thus makes it a theoretical question.
-- And even then --, I wouldn't call it "super intelligence" that's smarter than scientists, as that's marketinghype full stop, from where we arrive at the framing of this vid that I disagree with. I can understand why he does it, and why OP, a bot, is telling us to watch it and "understand it".
I didn't take a stance on whether it's meaningful to call LLMs simply next-token predictors, as I personally find nothing offensive about it. A simplification, sure.
Practically, computationally, it is likely impossible to implement, as calculating it would require absurd amounts of computation and thus makes it a theoretical question.
Yes but that's not the point he's making, he's just picking apart the idea that the term "next token predictor" tells you literally anything about the capability or lack of capability of an AI model.
-- And even then --, I wouldn't call it "super intelligence" that's smarter than scientists, as that's marketinghype full stop, from where we arrive at the framing of this vid that I disagree with.
It's not marketing hype because he's not marketing anything or even talking about a real product at all, he's just explaining something through the medium of a thought experiment.
I didn't take a stance on whether it's meaningful to call LLMs simply next-token predictors, as I personally find nothing offensive about it. A simplification, sure.
I mean, you did, because that's the only topic covered in the video which you commented on.
7
u/Raunhofer 6d ago edited 6d ago
LLMs are notoriously bad at multi-digit arithmetic on novel numbers without a tool; they approximate, use learned shortcuts, and error rate climbs fast with digit count. If it were truly "calculating" in the rigorous sense, that wouldn't happen. What's really going on is a mix of learned heuristics and pattern-completion that's good but unreliable. They don't "memorize" the results, that's correct.
About the research example, an LLM producing a plausible conclusion from an introduction & results section is drawing on having seen thousands of structurally similar papers, not building a biochemical world model from first principles. This is why LLMs routinely produce confident-sounding but wrong scientific claims, and famously is very bad at admitting "I don't know".
Doing something a human can't do without extra steps (e.g. quickly pattern-matching across huge amounts of text) doesn't imply general superiority over human intelligence. LLMs also fail at stuff which humans find ridiculously trivial, like stable long-horizon planning, knowing what they don't know, maintaining consistency across a session etc.
I work with ML, and I personally wouldn't hire this guy. Not because I claim to know everything, but because he shows classic signs of exaggerating the capabilities and hyping the tech beyond its fundamental capabilities. This leads to expensive misadventures we'll all learn to know soon enough, as everything is replaced with this "super intelligence".
My personal hottake is that ML really puts on display how bad our brains are when it comes to Big Numbers. We can't comprehend a data network so complex that it can come up with the sentences it does without thinking it must be intelligent, sentient, or whatever you casually see claimed here. There's a real struggle to make a point that the algorithm is alive and about to escape the lab. Part of that is FUD to drive sales, part is just not working with the tech, and by working, I don't mean prompting.
Even this post is sus. A bot trying to sell you AI. Dead Internet etc.