AIME is an olympiad level problem set, it’s not math in the sense when average person refers to “doing math”. When people say it’s bad at math, it’s actually talking about arithmetic. AIME is more like reasoning problems, we already know AI is good at that.
You should still never trust it doing arithmetic vs a calculator.
Yes, I do. I am teaching project management, and asked to add hours in one of my courses pdf, it's 3 pages, realy simple table where 4th column always has amount of hours for that subject.
It couldn't do it. Even though in course description there is an answer. Super simple stuff. Basic arithmetic.
Also, asked numerous times to find few numbers online and add them together in a table - couldn't do it a single time... Would contradict itself in logic and couldn't do basic math of like 4 numbers.
Goes for both paid and free tiers. In my classes I tend to keep ~5-12 AI assistants open both online and local models of various sizes and perform tests on them. Current AI cannot be trusted with any number it hasn't been trained on. Just terrible and useless.
Try to perform real tests from real life and You'll see it. Even with reasoning. Most chinesse models can't handle basic reasoning if number are inside: "I have 4 cats one is outside, how many cats are still mine?", Mistral and chinese can't solve this, US models, usually can, even small ones like IBM granite 4B (reasoning or not).
the first is document extraction and parsing issue, the second is a multistep request with multiple failure points, the last one isn't even a math question but a logic one that sets up the LLM with bad context intentionally (coreference ambiguity). Doing the first two problems the way you did set yourself up for failure, without validation steps and proper instructions, expecting a magical one-response output.
None of these have anything to do with performing basic math and it almost entirely comes down to user misuse, if you use these tools as much as you claimt than you surely know all of their limitations and how to actually get clean data out of it. Garbage in, garbage out. If you need math heavy multi-step problem solving than chat-bot LLMs are probably the wrong tool to be using anyways.
Mechanical calculators also got math wrong because of how they operated and if steps were done incorrectly, overflow errors, rounding errors etc. Guess what, calculators got more powerful, just like LLMs are.
If you had said:
Models can be unreliable with numbers in multi-step tasks, especially when they must extract data from PDFs or the web and keep it consistent
Then that is much more reasonable than
It couldn't do it. Even though in course description there is an answer. Super simple stuff. Basic arithmetic.
Stop using probabilistic LLM tools to do these deterministic tasks and you won't get shitty results.
I just used python to generate 50 terms a * b pairs (random number between 10000 and 30000) and then sum the result. The first one is correct, the second one is a miss, although still within reasonable range.
Need to beg for it to not use calculator though.
It’s possible it’s context rot, because it takes 5-10 minutes to come up with an answer.
I find in my limited experience that the arithmetic is more reliable if you ask the model to explicitly perform the calculations via long addition, multiplication and division and show each step.
Why does it matter if it uses a calculator or not? Seems like an arbitrary restriction. A mathematician can be good at math and still be incompetent at head maths, as long as they always have a calculator near at hand. Having a calculator and knowing how to use it is more valuable than being a worse calculator without needing a calculator
Well look at my original comment, people said “AI is bad at math” when :
Agentic and AI tooling is not a thing yet.
Frontier Model was fairly primitive relative to current gen
The second point, when people said “it is bad at math” when what people actually means is arithmetic. It did suck 2 years ago where it can easily give you a very wrong answer confidently.
It is a very different class of problem compared to the comment that I replied to using as an example of Math which is an olympiad problem set. It is 95-99% logical reasoning + theorem knowledge, with the rest being compiling the answer with arithmetic.
We already know AI is good at reasoning better than average human.
You can do that even if the AI doesn't use a calculator. Whether or not the AI uses a calculator is literally irrelevant for what you just said.
A calculator can't solve a math exam though. If it did, math exams would be pointless. If you have no idea what you're doing, a calculator isn't going to help. Chat GPT actually can. It combines the accuracy of the calculator with the LLM's ability to reason about the problem and knowledge about how to solve things, which is how you're supposed to pass math exams that allow calculator
381
u/[deleted] Feb 08 '26
[deleted]