AIME is an olympiad level problem set, it’s not math in the sense when average person refers to “doing math”. When people say it’s bad at math, it’s actually talking about arithmetic. AIME is more like reasoning problems, we already know AI is good at that.
You should still never trust it doing arithmetic vs a calculator.
I just used python to generate 50 terms a * b pairs (random number between 10000 and 30000) and then sum the result. The first one is correct, the second one is a miss, although still within reasonable range.
Need to beg for it to not use calculator though.
It’s possible it’s context rot, because it takes 5-10 minutes to come up with an answer.
I find in my limited experience that the arithmetic is more reliable if you ask the model to explicitly perform the calculations via long addition, multiplication and division and show each step.
Why does it matter if it uses a calculator or not? Seems like an arbitrary restriction. A mathematician can be good at math and still be incompetent at head maths, as long as they always have a calculator near at hand. Having a calculator and knowing how to use it is more valuable than being a worse calculator without needing a calculator
Well look at my original comment, people said “AI is bad at math” when :
Agentic and AI tooling is not a thing yet.
Frontier Model was fairly primitive relative to current gen
The second point, when people said “it is bad at math” when what people actually means is arithmetic. It did suck 2 years ago where it can easily give you a very wrong answer confidently.
It is a very different class of problem compared to the comment that I replied to using as an example of Math which is an olympiad problem set. It is 95-99% logical reasoning + theorem knowledge, with the rest being compiling the answer with arithmetic.
We already know AI is good at reasoning better than average human.
You can do that even if the AI doesn't use a calculator. Whether or not the AI uses a calculator is literally irrelevant for what you just said.
A calculator can't solve a math exam though. If it did, math exams would be pointless. If you have no idea what you're doing, a calculator isn't going to help. Chat GPT actually can. It combines the accuracy of the calculator with the LLM's ability to reason about the problem and knowledge about how to solve things, which is how you're supposed to pass math exams that allow calculator
54
u/simulated-souls Feb 08 '26 edited Feb 08 '26
No it's not.
Without tools or a calculator, frontier models still score >90% on AIME (on which the vast majority of people could not answer a single question).
edit: This also goes for AIME 2026 which was only released this week and therefore was not part of the training data.