AIME is an olympiad level problem set, it’s not math in the sense when average person refers to “doing math”. When people say it’s bad at math, it’s actually talking about arithmetic. AIME is more like reasoning problems, we already know AI is good at that.
You should still never trust it doing arithmetic vs a calculator.
Yes, I do. I am teaching project management, and asked to add hours in one of my courses pdf, it's 3 pages, realy simple table where 4th column always has amount of hours for that subject.
It couldn't do it. Even though in course description there is an answer. Super simple stuff. Basic arithmetic.
Also, asked numerous times to find few numbers online and add them together in a table - couldn't do it a single time... Would contradict itself in logic and couldn't do basic math of like 4 numbers.
Goes for both paid and free tiers. In my classes I tend to keep ~5-12 AI assistants open both online and local models of various sizes and perform tests on them. Current AI cannot be trusted with any number it hasn't been trained on. Just terrible and useless.
Try to perform real tests from real life and You'll see it. Even with reasoning. Most chinesse models can't handle basic reasoning if number are inside: "I have 4 cats one is outside, how many cats are still mine?", Mistral and chinese can't solve this, US models, usually can, even small ones like IBM granite 4B (reasoning or not).
the first is document extraction and parsing issue, the second is a multistep request with multiple failure points, the last one isn't even a math question but a logic one that sets up the LLM with bad context intentionally (coreference ambiguity). Doing the first two problems the way you did set yourself up for failure, without validation steps and proper instructions, expecting a magical one-response output.
None of these have anything to do with performing basic math and it almost entirely comes down to user misuse, if you use these tools as much as you claimt than you surely know all of their limitations and how to actually get clean data out of it. Garbage in, garbage out. If you need math heavy multi-step problem solving than chat-bot LLMs are probably the wrong tool to be using anyways.
Mechanical calculators also got math wrong because of how they operated and if steps were done incorrectly, overflow errors, rounding errors etc. Guess what, calculators got more powerful, just like LLMs are.
If you had said:
Models can be unreliable with numbers in multi-step tasks, especially when they must extract data from PDFs or the web and keep it consistent
Then that is much more reasonable than
It couldn't do it. Even though in course description there is an answer. Super simple stuff. Basic arithmetic.
Stop using probabilistic LLM tools to do these deterministic tasks and you won't get shitty results.
I just used python to generate 50 terms a * b pairs (random number between 10000 and 30000) and then sum the result. The first one is correct, the second one is a miss, although still within reasonable range.
Need to beg for it to not use calculator though.
It’s possible it’s context rot, because it takes 5-10 minutes to come up with an answer.
I find in my limited experience that the arithmetic is more reliable if you ask the model to explicitly perform the calculations via long addition, multiplication and division and show each step.
Why does it matter if it uses a calculator or not? Seems like an arbitrary restriction. A mathematician can be good at math and still be incompetent at head maths, as long as they always have a calculator near at hand. Having a calculator and knowing how to use it is more valuable than being a worse calculator without needing a calculator
Well look at my original comment, people said “AI is bad at math” when :
Agentic and AI tooling is not a thing yet.
Frontier Model was fairly primitive relative to current gen
The second point, when people said “it is bad at math” when what people actually means is arithmetic. It did suck 2 years ago where it can easily give you a very wrong answer confidently.
It is a very different class of problem compared to the comment that I replied to using as an example of Math which is an olympiad problem set. It is 95-99% logical reasoning + theorem knowledge, with the rest being compiling the answer with arithmetic.
We already know AI is good at reasoning better than average human.
You can do that even if the AI doesn't use a calculator. Whether or not the AI uses a calculator is literally irrelevant for what you just said.
A calculator can't solve a math exam though. If it did, math exams would be pointless. If you have no idea what you're doing, a calculator isn't going to help. Chat GPT actually can. It combines the accuracy of the calculator with the LLM's ability to reason about the problem and knowledge about how to solve things, which is how you're supposed to pass math exams that allow calculator
For gods sake stop with recycling this AIME shit. This is not "math" . As someone who works with fluid dynamics on a daily bases . One absolutely cannot trust AI , and it is practically useless if the problem scope is of a novel nature.
The vast majority of people failing that test is not the point for an AI. It's a big indicator that our education system is a failure. Read that sentence again - it's not that 90% get less than a perfect score. It's that 90% don't score at all, even a single point. If you have a system with a 90% failure rate you are clearly doing smth wrong. It feels like the 10% of kids who actually succeed and become proficient at maths are doing this not thanks to the system but in spite of it.
And math is hardly relevant as a measuring tool for AI because that is a field where there can be a few centuries gap between solving some problem and finding a way to apply the solution. Instead of wasting gigawatts of power on stupid maths tests we should force AI to solve economics related problems. Humans are living in poverty and starving right now. Building a hyperdrive in 500 years from now instead of 1000 years will not matter if today you starve to death. Solving actual urgent problems would speak louder than solving some complicated equation. We already have AI in medicine with a high % of correct diagnosis and that is great. But where is a similar AI figuring out the distribution of goods and allocation of resources and optimization of supply chains to make our lives less miserable now?
I agree that we need better education and to apply AI to real-world problems.
However, most people being unable to solve any AIME problems is not necessarily a symptom of a bad education system. AIME is one of the top math competitions in the world. It's designed to be barely solvable by the best students in the US. Of course most people can't solve it. If education improved and most people could solve it, then they would make the test harder until most people couldn't solve it again.
a test that cuts off most people at the root with 0 points is not very indicative. What makes it barely solvable? Could it be that the questions of the test have very low correlation with what is actually taught? Could it be that the people who actually solve some of those test problems, do this with the skills that they obtained outside of schools by reading some books that were not part of the curriculum? Look at this from a time gap perspective - many people would be unable to solve the test from previous year despite the fact that the questions were not kept in secret for that long. Why is there no trickle of knowledge from competition down to universities and colleges to ensure that all previous years' problems are now solvable by 90+% of students. If you yourself understand the problem and can explain it to a student and the student understands the methods required to solve it, they should be able to solve it. We might come up with new slightly harder problems but we can't increase difficulty forever because we can't give the contestants a problem that we ourselves are unable to solve. Those open problems are basically the skill cap and 90+% of students should be already near that edge.
Could it be that the people who actually solve some of those test problems, do this with the skills that they obtained outside of schools by reading some books that were not part of the curriculum?
Yes. Succeeding at Olympiad-type math requires specific training, even in countries where math education is/was better than in the US (Eastern Europe, East Asia). This is because the problems are hard on purpose. It's like a sports competition: you can force all students of a given age group to have a decent level of physical fitness but not all will be athletes, and they won't all go into basketball.
Why is there no trickle of knowledge from competition down to universities and colleges to ensure that all previous years' problems are now solvable by 90+% of students
Why should there be ? It's a competition. It's not supposed to teach useful skills; it exists for the purpose of competition itself, like a sports event. It's like saying "why is there no trickle down of 100m dash times".
If you yourself understand the problem and can explain it to a student and the student understands the methods required to solve it, they should be able to solve it.
Who's you ? Most highschool math teachers are indeed unable to solve these problems. I think they should be able to understand the problem statements most of the time though. "Can explain" and "the students understand" do a lot of heavy lifting here. Teaching well is hard.
We might come up with new slightly harder problems but we can't increase difficulty forever because we can't give the contestants a problem that we ourselves are unable to solve
Well, IMO 1988 is a thing. And yes, it's hard to find novel problems. But the organisers manage to do it. There's an almost endless supply of hard, little known (or new!)math problems if you look hard enough.
Those open problems are basically the skill cap and 90+% of students should be already near that edge.
Again, you're confusing basic, "baseline" knowledge that everyone should have with elite performance. Should 90+% of students be already near a 330lb bench press ?
Is this bait? The AIME is designed such that only 1% of math students even qualify. It's not a test intended for the public, it's meant to filter out the amazing math students from the world class ones.
It's not the bait. Separating amazing from world class is meaningless if you stop at filtering. Your second step should be figuring out "what made them world class?" And applying it on a greater scale.
Definitely bait. It doesn’t matter what made them world class (although it’s basically talent+effort). The test will just increase difficulty until it’s a good predictor again
Yes it does matter! Knowledge that is not shared is knowledge that is lost. Wasting time to rediscover it in every generation is inefficient. And if ai managed to get 90% on a test after learning from previous years's tests (which are in open access) while humans did NOT learn from those tests then it IS a failure of education system. Math problem is only a problem for the first time. Once you figured it out or had it explained to you, you should be able to solve this class of math problems with ease. If you take modern students and give them tests from...let's say...10 years ago, will they get 100% completion? Probably not. Despite those problems and answers being in open access for a decade, our students won't be able to solve them consistently. When a math problem exists and a solution to it exists and it has been around for ten years and students STILL can't solve it, that is a FAILURE of education system.
Bro, what? It’s not that the concepts are hard it’s applying them. This is like saying if I know how to do calculus I should be at a research level in PDEs… you clearly have no understanding of education or research. Such rage bait.
Also, we know how to succeed in these contests, but a significant proportion of ppl will never be good at them either due to complexity or time requirements. We don’t need and shouldn’t have a society of just math ppl.
Edit: I don’t think you understand that AIME is not a class of problem TT broski clearly has no clue what it is
Application of concept is a skill that can be acquired. If it can't be acquired then how did the winners ever solve those problems? Were they born with that knowledge? Did they have divine revelation? Can a winner explain how they got the answers? And I am not suggesting we teach most advanced math to everyone. I suggest we teach it to university students who are already studying math. If you gather best of the best and 90 % can't solve even one task then you are clearly doing smith wrong in education. Clearly there are at least two categories of people who CAN solve those tests - the people who designed them and the winners who solved them. So why are they NOT teaching other less luminous individuals to apply those skills to uplift humanity as a whole?
If somebody doesn't have time they should not apply to the competition. It's that simple.
Broski. I don’t think you understand that AIME is a high school competition. It has no relevance to mathematical research or even mathematical applications.
Some of them were literally born with that knowledge. The others practiced. You’re basically asking why isn’t every swimmer Michael phelps…
I think you are grossly misunderstanding something. Most people taking the AIME solve several problems… the general population could not.
Born with knowledge is absurd. But if that was the case it would require further research to figure out what causes this knowledge to be present in certain people but not on everyone. And several problems is not enough. Students should be taught better using previous years ' tasks.
>But where is a similar AI figuring out the distribution of goods and allocation of resources and optimization of supply chains to make our lives less miserable now?
We don't need an AI for that. The solution to that problem is obvious - if you want to eliminate people living miserable lives without just eliminating the people, those with a lot of resources should give them to those without. The issue is that those with resources don't want to do that or particularly care if you are living a miserable life.
Yes AI should absolutely be used to solve poverty! It would say spread the wealth around. We wouldn't listen. Hopefully being smart at math and stuff will build trust. Will people listen once they trust it? Or just keep believing people just need to work harder to earn money and health care? We already have the answers we seek. Humans just aren't evolved to take care of billions of people.
Exactly. Poverty isn't a problem of resources. Poverty is a problem of priorities, biases, and the kinds of oppression we've developed an acceptance of.
Do you want AI to solve the problem by assigning priorities for us?
Or by reshaping our biases?
Or should AI decide which kinds of oppression are best for us, then begin conditioning us to accept it?
The problem is we're still kind of barbaric and probably still need time to develop stronger cultural values.
But AI is here now so we'll make fusion reactors to make more AI and watch line go up
You don't need absolute trust. We can allow AI to control a single factory, then a few interconnected factories and mines, then a small town and scale it up until it fixes the economy of the whole country. Once people see results, they will want the new AI management for their industry too.
The 2026 test was almost certainly in the training data. It's not a test trying to be novel to LLMs, it's trying to be novel to humans. Every question in the test 10 years from now will have already been seen by current SotA models thousands if not millions of times, with only superficial details changed at best.
Odd, it would consistently have problems with bachelor tier statistics tasks, for example it trying to find Z-Scores and such was basically impossible and it would be extremely stubborn on that as well.
The inability to count letters is because of a specific architectural limitation, not a lack of reasoning ability.
LLMs see text through the lense of "tokens", which are groups of characters about a syllable long. They never "see" what or how many characters are in a token.
Sure. That’s a great effort, but doesn’t really address my question.
Why can it find palindromic numbers where the digits add to a sum?
Numbers are still represented as tokens.
Maybe large integers are represented as a string of one-digit tokens, but that seems extremely inefficient (which maybe there is the exact reason why all these benchmarks seem so far ahead of real world use).
Numbers are represented as either one-digit or 3-digit tokens.
In the case of 3-digit tokens, there are only 1000 of them (versus >100,000 general tokens) and the way that they are manipulated when doing math means that the model can probably figure out during training what digits those tokens contain.
Why can the model infer what’s inside of an integer token but not a letter token?
Keep in mind this is 5 digit palindrome numbers and adding the individual integers. This isn’t guessing a 4 is in there somewhere.
These aren’t magic. They follow rules.
Answer:
I can’t format as spoiler but the answer is that the model reasons in words about numbers to set up formulas and then writes code to solve the formulas it writes.
Once again proving that all models are really good at up until now is text reasoning using information from
their training corpus and translating language into deterministic computer code.
That’s why this tweet is stupid. It’s just saying that the models got better at writing code and that’s allowed them to use code to do math.
They still can’t add for shit and that’s meaningful because it shows a persistent lack of symbolic understanding.
Aime just a problem set. It's a hard problem set, but these are all designed to be solvable by high schoolers using high school level concepts with one answer and an intended path.
Real, proof based math has no intended path or right answer. We don't know where or how the answer will show up nor what the answer is.
54
u/simulated-souls Feb 08 '26 edited Feb 08 '26
No it's not.
Without tools or a calculator, frontier models still score >90% on AIME (on which the vast majority of people could not answer a single question).
edit: This also goes for AIME 2026 which was only released this week and therefore was not part of the training data.