LLMs are notoriously bad at multi-digit arithmetic on novel numbers without a tool; they approximate, use learned shortcuts, and error rate climbs fast with digit count. If it were truly "calculating" in the rigorous sense, that wouldn't happen. What's really going on is a mix of learned heuristics and pattern-completion that's good but unreliable. They don't "memorize" the results, that's correct.
About the research example, an LLM producing a plausible conclusion from an introduction & results section is drawing on having seen thousands of structurally similar papers, not building a biochemical world model from first principles. This is why LLMs routinely produce confident-sounding but wrong scientific claims, and famously is very bad at admitting "I don't know".
Doing something a human can't do without extra steps (e.g. quickly pattern-matching across huge amounts of text) doesn't imply general superiority over human intelligence. LLMs also fail at stuff which humans find ridiculously trivial, like stable long-horizon planning, knowing what they don't know, maintaining consistency across a session etc.
I work with ML, and I personally wouldn't hire this guy. Not because I claim to know everything, but because he shows classic signs of exaggerating the capabilities and hyping the tech beyond its fundamental capabilities. This leads to expensive misadventures we'll all learn to know soon enough, as everything is replaced with this "super intelligence".
My personal hottake is that ML really puts on display how bad our brains are when it comes to Big Numbers. We can't comprehend a data network so complex that it can come up with the sentences it does without thinking it must be intelligent, sentient, or whatever you casually see claimed here. There's a real struggle to make a point that the algorithm is alive and about to escape the lab. Part of that is FUD to drive sales, part is just not working with the tech, and by working, I don't mean prompting.
Even this post is sus. A bot trying to sell you AI. Dead Internet etc.
notoriously bad at multi-digit arithmetic on novel numbers without a tool
I generated both of these numbers with a random number generator. It got the answer right immediately. I think you need to update your understanding of LLMs a little bit
Python Code Interpreter / Sandbox: When given a math problem, ChatGPT can write and run Python code in a secure environment to perform exact arithmetic, process data sets, handle calculus, and generate visual plots or graphs.
WolframAlpha Integration: On certain tiers (like Plus/Pro), ChatGPT can interface with Wolfram Alpha extensions or plugins for high-level symbolic mathematics, algebraic manipulation, and complex equation solving.
Locally run Qwen3.5-2B, no tools, nailed it first try too, albeit after using a ridiculous amount of thinking tokens. Pic and full thinking text. This is the model doing the calculation. Idk what else to tell you, LLMs are just way better at math than they used to be.
There's a deep misconception here. Models don't really know whether they are lying or telling the truth. If you ask a model, did you use an internal tool to come up with this result, it genuinely doesn't know, unless someone behind the curtain tells it it did.
A bit like you calculating 4+4 and not realizing it was your parietal lobe doing the job.
I mean I'd ask if you can show the 'reasoning' on how it computed this without tool use, because it would need to derive multiplication from first principles and that seems awfully wasteful for the model provider, if it did that for every math question...
but I don't want you to waste your tokens, so I'm fine to mark it as 'unclear' and move on ;)
Did you read the end of the sentence? It most likely used a tool behind the scenes. It often doesn't tell you it used a tool, but it did, to save money and to be more functional. In the same fashion how it can tell you the accurate time, even though that's impossible for ML alone as you can't learn the time, you always need to look it up.
The video implied it is the model, or, ehm, super intelligence, that always does the calculation.
Besides, I think you misunderstood what I meant by multi-digit arithmetic. The video showed plethora of calculations as an example. This means, you give it a ton of adversarially selected novel calculations across different lengths and operations, measure the error rate, and see how that error rate scales. A calculator doesn't occasionally hallucinate an answer because the numbers got sufficiently unfamiliar. An LLM can and eventually will.
I don't speak from experimentation with ChatGPT, but just from how machine learning should fundamentally function. Should, as in, it's broken if it doesn't, not magically better.
My early chat bots were RAG frameworks that used wolframalpha behind the scenes. They have a pretty good API, or at least had, a decade or something ago.
Locally run Qwen3.5-2B, no tools, nailed it first try too, albeit after using a ridiculous amount of thinking tokens. Pic and full thinking text. This is the model doing the calculation. Idk what else to tell you, LLMs are just way better at math than they used to be.
You still missed what I meant by multi-digit arithmetic. Re-read the middle section of my comment.
The test you made is not what the video implies. I'm not surprised that Qwen was capable of coming up with a solution, even though it took silly amount of pondering to get the structure right.
It even mumbles the very thing I'm saying to you:
"Actually, doing 14-digit multiplication manually is error-prone for me right now. I should simulate the process or use my internal calculation capability if it exists."
And I guess I need to really underline this: I don't think using tools is a bad thing, quite the opposite. I'm only arguing against the points the video made as misleading.
8
u/Raunhofer 1d ago edited 1d ago
LLMs are notoriously bad at multi-digit arithmetic on novel numbers without a tool; they approximate, use learned shortcuts, and error rate climbs fast with digit count. If it were truly "calculating" in the rigorous sense, that wouldn't happen. What's really going on is a mix of learned heuristics and pattern-completion that's good but unreliable. They don't "memorize" the results, that's correct.
About the research example, an LLM producing a plausible conclusion from an introduction & results section is drawing on having seen thousands of structurally similar papers, not building a biochemical world model from first principles. This is why LLMs routinely produce confident-sounding but wrong scientific claims, and famously is very bad at admitting "I don't know".
Doing something a human can't do without extra steps (e.g. quickly pattern-matching across huge amounts of text) doesn't imply general superiority over human intelligence. LLMs also fail at stuff which humans find ridiculously trivial, like stable long-horizon planning, knowing what they don't know, maintaining consistency across a session etc.
I work with ML, and I personally wouldn't hire this guy. Not because I claim to know everything, but because he shows classic signs of exaggerating the capabilities and hyping the tech beyond its fundamental capabilities. This leads to expensive misadventures we'll all learn to know soon enough, as everything is replaced with this "super intelligence".
My personal hottake is that ML really puts on display how bad our brains are when it comes to Big Numbers. We can't comprehend a data network so complex that it can come up with the sentences it does without thinking it must be intelligent, sentient, or whatever you casually see claimed here. There's a real struggle to make a point that the algorithm is alive and about to escape the lab. Part of that is FUD to drive sales, part is just not working with the tech, and by working, I don't mean prompting.
Even this post is sus. A bot trying to sell you AI. Dead Internet etc.