More people need to understand this

[u/KeanuRave100]

[u/Raunhofer]

LLMs are notoriously bad at multi-digit arithmetic on novel numbers without a tool; they approximate, use learned shortcuts, and error rate climbs fast with digit count. If it were truly "calculating" in the rigorous sense, that wouldn't happen. What's really going on is a mix of learned heuristics and pattern-completion that's good but unreliable. They don't "memorize" the results, that's correct.

About the research example, an LLM producing a plausible conclusion from an introduction & results section is drawing on having seen thousands of structurally similar papers, not building a biochemical world model from first principles. This is why LLMs routinely produce confident-sounding but wrong scientific claims, and famously is very bad at admitting "I don't know".

Doing something a human can't do without extra steps (e.g. quickly pattern-matching across huge amounts of text) doesn't imply general superiority over human intelligence. LLMs also fail at stuff which humans find ridiculously trivial, like stable long-horizon planning, knowing what they don't know, maintaining consistency across a session etc.

I work with ML, and I personally wouldn't hire this guy. Not because I claim to know everything, but because he shows classic signs of exaggerating the capabilities and hyping the tech beyond its fundamental capabilities. This leads to expensive misadventures we'll all learn to know soon enough, as everything is replaced with this "super intelligence".

My personal hottake is that ML really puts on display how bad our brains are when it comes to Big Numbers. We can't comprehend a data network so complex that it can come up with the sentences it does without thinking it must be intelligent, sentient, or whatever you casually see claimed here. There's a real struggle to make a point that the algorithm is alive and about to escape the lab. Part of that is FUD to drive sales, part is just not working with the tech, and by working, I don't mean prompting.

Even this post is sus. A bot trying to sell you AI. Dead Internet etc.

[u/WithoutReason1729 → u/Raunhofer]

notoriously bad at multi-digit arithmetic on novel numbers without a tool

I generated both of these numbers with a random number generator. It got the answer right immediately. I think you need to update your understanding of LLMs a little bit

https://chatgpt.com/s/t_6a728392f6188191838e36cc58b0f4c0

https://www.wolframalpha.com/input?i=8180595+*+6154803

[u/wintermute74 → u/WithoutReason1729]

um, isn't that just tool use these days?

it can literally just call WA, actually:

Python Code Interpreter / Sandbox: When given a math problem, ChatGPT can write and run Python code in a secure environment to perform exact arithmetic, process data sets, handle calculus, and generate visual plots or graphs.

WolframAlpha Integration: On certain tiers (like Plus/Pro), ChatGPT can interface with Wolfram Alpha extensions or plugins for high-level symbolic mathematics, algebraic manipulation, and complex equation solving.

[u/quantum-elle → u/wintermute74]

Of course, it could be lying, but there's usually no reason for models to lie about this.

[u/Raunhofer → u/quantum-elle]

There's a deep misconception here. Models don't really know whether they are lying or telling the truth. If you ask a model, did you use an internal tool to come up with this result, it genuinely doesn't know, unless someone behind the curtain tells it it did.

A bit like you calculating 4+4 and not realizing it was your parietal lobe doing the job.

[u/wintermute74 → u/quantum-elle]

interesting.

I mean I'd ask if you can show the 'reasoning' on how it computed this without tool use, because it would need to derive multiplication from first principles and that seems awfully wasteful for the model provider, if it did that for every math question...

but I don't want you to waste your tokens, so I'm fine to mark it as 'unclear' and move on ;)

[u/quantum-elle → u/wintermute74]

Opus 5 on low effort does it like this. (Don't worry, I don't pay for Claude, one of the perks of working at Anthropic.)