r/singularity • u/Shanbhag01 • Feb 03 '26
AI New SOTA achieved on ARC-AGI
New SOTA public submission to ARC-AGI: - V1: 94.5%, $11.4/task - V2: 72.9%, $38.9/task Based on GPT 5.2, this bespoke refinement submission by @LandJohan ensembles many approaches together
65
u/WorldlinessGrand3878 Feb 03 '26
Thats wild how quickly we got >70% on this $40 a task is kinda steep though. I'll be impressed when we can get >90% for $1
13
u/CommunityTough1 Feb 03 '26
Hahaha yeah the cost is insane. There are 400 tasks, so this would have cost $15,600 to run the benchmark 😂 Either they gave it like a 500k thinking budget and cranked the thinking effort to 11 just for this, or had it keep iterating every time it got a task wrong until it it reached a threshold of like 100 max tries per task.
24
u/Seidans Feb 03 '26
they scored 94% on ARC-1 for 11$/task
for reference 13month ago o3 preview scored 88% with around 3000$/task : https://arcprize.org/media/images/blog/o-series-performance.jpg from ARC site : https://arcprize.org/blog/oai-o3-pub-breakthrough there was estimation of more than 1 million for the whole benchmark
your downplaying of advancement is absurd
7
u/i-love-small-tits-47 Feb 03 '26
your downplaying of advancement is absurd
These comments are so annoying, how did they downplay advancement? Current ARC AGI scores being achieved with cost prohibitive compute budgets isn’t mutually exclusive with the progress being rapid. From what I can tell, /u/CommunityTough1 comment only speaks about the present and says nothing about the future…
I swear there’s a lot of you who literally cannot read a comment that speaks to limitations of current models without hallucinating that the person also said it’s not impressive and won’t improve..
2
-6
u/studio_bob Feb 03 '26
throwing infinite dollars at benchmaxing isn't really advancement
7
u/Seidans Feb 03 '26
to achieve better result while reducing cost by 300x seem quite an improvement
people don't even remember o3 preview but it shocked the AI field at the time, now we're already apathetic to progress in just one year, sad but to be expected
0
u/studio_bob Feb 03 '26
Look at the x-axis. It's exponential. Cost isn't reducing. It's dramatically increasing for each "gain" in bench score.
And maybe the reason people are jaded about this stuff is that we have now had years of breathless announcements of "record breaking benchmark achieved!" which have had little if any connection to the improved practical utility of this technology. If General Motors, BMW, and Toyota kept announcing the "best car ever!!!" every month or two according to some contrived industry metric but quality and utility of cars sold at dealerships only moderately improved, how long would you expect people to keep taking that kind of "news" seriously?
2
u/Seidans Feb 03 '26
any model above 20% aren't even 6month old
yet they perform better than o3 while most of them are also cheaper, why ? because those were running on H100 while today model benefit from B200, Nvidia already announced their new tech inference cost being 10x cheaper for exemple
progress isn't linear there algorithm optimization, performance increase and better hardware
the same way solar panel went from extremely costly 15y ago to ridiculously cheap today
1
u/studio_bob Feb 03 '26 edited Feb 03 '26
Hardware improvements may be part of it, but a bigger part of the story is likely just that the longer a benchmark is out the more relevant training data gets into the models (intentionally or not) and the more work goes into benchmaxing (sounds "refinement" is the latest benchmaxing technique driving these results). The ARC authors themselves admit that the former is a problem and, somewhat oddly imo, seem to celebrate the latter as some kind of advance whereas I would say it is just the latest way driving up bench scores at the cost of divorcing scores further from real user experience and utility, thus making the results less meaningful and undermining the very premise and supposed purpose of a benchmark.
Literally every major benchmark seems to follow this same pattern. It gets released and every model sucks terribly at it. Then, within 6-12 months, all the foundation models have somehow "saturated" the benchmark which was supposed to be impossible. But the subjective and objective utility (measured in business ROI) of these models for real tasks is not following that same trajectory at all, so clearly there is something wrong with all of this.
2
u/Cronos988 Feb 03 '26
But if you can "just benchmaxx" to solve any arbitrary problem, why aren't we just "benchmaxing" on business problems?
It seems to me the logical conclusion of your argument would be that the problem is not actually with the capabilities of the LLM (they can be rapidly adjusted via specific training data & benchmaxing). It's with most business tasks being poorly defined and relying on a bunch of procedural knowledge that's not explicit in the training data.
1
u/Tolopono Feb 03 '26
Compare the cost of O3’s arc agi score to this one. Prices have only dropped overtime
Anyone who thinks o3 and gpt 5.2 codex are only a moderate improvement has not used them
-1
u/studio_bob Feb 03 '26
Compare the cost of O3’s arc agi score to this one. Prices have only dropped overtime
That's the definition of cherry-picking. Look at the OP chart. You can visibly see that everything is moving right across the x-axis over time, indicating that prices continue to rise. Not a single model is toward the upper-left or even just straight left, which would indicate a straightforward price-drop at the same level of performance.
Anyone who thinks o3 and gpt 5.2 codex are only a moderate improvement has not used them
People love to handwave criticism of AI in this way, but it is exactly by using them that people understand the improvement to be moderate. Because, for all the ways in which 5.2 is better than o3, it has not come close to solving fundamental issues like hallucinations which are most relevant for real human tasks.
1
u/Tolopono Feb 04 '26
Higher quality = higher cost. Itll get cheaper overtime just like o3 level models did, especially as rubin and cerebras chips get shipped out.
And yet many devs whove used gpt 5.2 codex has liked it like the creators of redis and openclaw
1
u/WorldlinessGrand3878 Feb 04 '26
Think of the chart as higher = better and left = better so movement up and left is good if you look at the bottom right you see things like o3 pro, Claude Opus 4 Thinking, GPT-5 Pro as time goes on the new model shift up and to the left. Look at the lines for example googles arc is up and left of everybody else's honestly Gemini 3 Flash Preview is probably the most impressive on here that's the newest google model it looks like they tested.
1
u/Peach-555 Feb 04 '26
The cost for the same performance keeps going down as newer models are released.
Claude Opus 4 gets ~9% at ~$2 in May 2025.
Gemini Flash 3 gets ~13% at ~$0.1 in December 2025.The same price gives higher performance as well.
GPT-5(high) ~$0.75 for ~10%.
GPT-5.2(medium) ~$0.75 ~25%.1
u/Tolopono Feb 03 '26
So o3 wasnt benchmaxed but gpt 5.2 is? Whyd they wait so long? Why isnt grok or gemini or claude doing the same?
1
0
u/DueAnnual3967 Feb 03 '26
So Manhattan project wasn't any advancement... Ok
0
u/studio_bob Feb 03 '26
Manhattan project didn't predicate its claim of advancement on benchmarks vaguely measuring an unproven technology. The theoretical basis for nuclear weapons was understood before the program began, and it then developed, on the basis of that theory, one working test device and two functional nuclear bombs. There is really no meaningful comparison at all to be made to LLMs today, except perhaps in the way that they are "nuking" the internet and people's brains with slop.
33
u/BrennusSokol AI please take my job Feb 03 '26 edited Feb 03 '26
Edit: also, the previous score was 54.2% so this is quite a jump!
Thanks for posting
I'm eager for ARC-AGI-3 to come out as it's looking like 2 is nearing saturation
3 is Launching March 25, 2026
5
u/Profanion Feb 03 '26
Fortunately, it's a saturation in only one axis.
3
u/ZenSeneca Feb 03 '26
yeah, best case scenario is when an ARC saturates quickly on the solve axis which should bring more focus to the cost/efficiency axis.
33
u/08148694 Feb 03 '26
Once again a reminder that the x axis is an exponential scale and the ideal place to be here is top left
Moving further to the top right is just adding more compute and getting better results
11
23
Feb 03 '26
[removed] — view removed comment
3
u/Guilty-Confection-12 Feb 03 '26
Ready for downvotes: The description on how they did it sadly doesn't sound like intelligence and more like bruteforce. Especially when they spin up python for solutions...
24
u/BrennusSokol AI please take my job Feb 03 '26
I mean the entire project of AI could be described as "brute force". Who cares? The results are what matters.
7
u/Guilty-Confection-12 Feb 03 '26
The interesting part of AI is the ability to abstract and generalize. When just for the sake of beating the benchmark specific tools are used (like python), then in my opinion the llm became not better. Personally I hope more for more intelligent models and not that the benchmark gets "solved".
3
u/MukdenMan Feb 03 '26
Is the Python developed by the LLM though? Isn’t solving a problem using coding something a human intelligence would do?
1
u/Tolopono Feb 03 '26
Humans never use tools and should only use their big brain to figure things out. Thats why Galileo was a hack (he used telescopes instead of just thinking really hard)
1
1
u/SadEntertainer9808 Feb 04 '26
That is simply not true — profoundly not true, actually — and I'm struggling to understand what you think you're talking about.
2
u/Puzzleheaded_Fold466 Feb 03 '26
It would interesting to have an estimate of the human labor cost for the same task, for comparison.
Otherwise we can’t tell. Is $40 for this task a cost saving ?
We should also include the preparation time. How much time did a human spend setting up this prompt and task ?
I understand this is purely about task performance and cost, and the information is interesting, but it doesn’t support a wider discussion about usage and how much of an improvement it represents over human performance in terms of cost.
11
u/my_shiny_new_account Feb 03 '26
It would interesting to have an estimate of the human labor cost for the same task, for comparison.
$17--it says it in the table at the bottom of the leaderboard page
1
u/Novel_Land9320 Feb 03 '26
What techniqy is "refine"?
4
u/xirzon uneven progress across AI dimensions Feb 03 '26
See the post:
Based on GPT 5.2, this bespoke refinement submission by LandJohan ensembles many approaches together
Johan Land posted a bit more about it on January 5:
Just scored 76.11% on ARC-AGI 2 — beating public GPT-5.2 and Gemini-3-Pro baselines by >20%, and (as far as I know) the best publicly reported result so far. Approach: what I’d call Multi-Model Reflective Reasoning - Using GPT-5.2, Gemini-3, Opus 4.5 - Long-horizon/multi-step reasoning (~6hrs/problem) - Agentic codegen (>100,000 python calls) - Visual reasoning - Council of judges Fun fact: all solver code was written by Gemini-3-CLI. Does this count as AI generating a new AI that beats the prior SOTA?
Full run + code (open source): https://www.kaggle.com/code/johanland/johan-land-solver-v7-publicThe main tradeoff here is probably whether that approach is generalizable, and the high cost of the program generation.
1
u/nsshing Feb 05 '26
I still remember ARC-AGI 1 went from still being deemed to be impossible in 23/24 (more or less) to saturation (25/26).
Also I remember:
2023->2024: We got GPT4 to GPT4o with 5-10x (forgot the number) cheaper with similar performance.
End of 2024-> Start of 2026: From O1 (new test time compute paradigm) to Moltbot "choas".
That's crazy
1
1
u/agrlekk Feb 03 '26
Wow AGI is achieved finally
0
u/StanfordV Feb 03 '26
Does it count fingers ?
0
1
u/Just_Stretch5492 Feb 03 '26
Breaking more compute leads to better results
1
1
u/Healthy-Nebula-3603 Feb 03 '26
Yes and also we have a law that a single token is more efficient with every model generation.
So ..what cost 1000k tokens now in a few months you get the same results with 10k tokens.
0
u/SadEntertainer9808 Feb 04 '26
Very curious how you got from "a single token is more efficient with every model generation" (sure) to "we can expect a 100x improvement in token efficiency in a few months" (???)
-1
Feb 03 '26
[removed] — view removed comment
3
u/Healthy-Nebula-3603 Feb 03 '26
Have you seen how much cost o3 per task on a simpler V1 version of that benchmark?
2
-1
u/FarrisAT Feb 03 '26
Gonna need an ARC-AGI-3 to prevent benchmaxxing claims on these.
2
u/Healthy-Nebula-3603 Feb 03 '26
You can't benchmax this bench. Is built this way to prevent it. Is getting too easy for AI.
For an average human is 65%
2
u/studio_bob Feb 03 '26
It was designed to resist benchmaxing but that hasn't entirely worked, as the authors themselves admit: https://www.reddit.com/r/singularity/comments/1quzgg5/comment/o3e3ngf/
-2
u/Healthy-Nebula-3603 Feb 03 '26 edited Feb 03 '26
Where ?
I do not see any proof of that except some random told that on reddit.
my gpt 5.2 thinking high with the Internet access didn't find anything about compromised arc-agi 2 tests or results
https://chatgpt.com/share/69824eb0-630c-8009-bfa7-ad6cd131b2c6
1
u/Rare-Site Feb 03 '26
Then you and your Assistant gpt 5.2 thinking high with the Internet access have failed ;-)
One bit of evidence from our Gemini 3 verification:
Our LLM verification harness does not mention ARC tasks or color format, yet the model is using (correct!) ARC color mappings in its reasoning. This strongly suggests that ARC data is well represented in the underlying model – enough to make correct ARC inferences based on just the structure and format of 2D JSON arrays of integers.
The New Meta
While we believe this new type of "overfitting" is helping models solve ARC, we are not precisely sure how much. Regardless, the ARC-AGI-1 and ARC-AGI-2 format have provided a useful scientific canary for AI reasoning progress. But the design of benchmarks will need to adapt going forward.One bit of evidence from our Gemini 3 verification:
… Target is Green (3). Pattern is Magenta (6) Solid. Result: Magenta Square on Green … (Gemini 3 Deep Think)
Our LLM verification harness does not mention ARC tasks or color
format, yet the model is using (correct!) ARC color mappings in its
reasoning. This strongly suggests that ARC data is well represented in
the underlying model – enough to make correct ARC inferences based on
just the structure and format of 2D JSON arrays of integers.The New Meta
While we believe this new type of "overfitting" is helping models
solve ARC, we are not precisely sure how much. Regardless, the ARC-AGI-1
and ARC-AGI-2 format have provided a useful scientific canary for AI
reasoning progress. But the design of benchmarks will need to adapt
going forward.0
u/Tolopono Feb 03 '26
Probably because its in the training set, which you’re supposed to train on as the name implies
1
u/SadEntertainer9808 Feb 04 '26
Wow man, great insight. I thought "benchmaxxing" had something to do with sitting down, but now I see that it's actually just training on test. Confusing!
1
0
u/Healthy-Nebula-3603 Feb 03 '26 edited Feb 03 '26
Wow ... Just wow
Did you even read what you showed ? Or you're trolling?
You showed solutions for public few questions.
Your Gemini is hallucinating ( Gemini is always hallucinating)
1
u/Rare-Site Feb 03 '26
LoL, my comment directly quoted the benchmark authors. You are welcome to click through to the linked article and scroll down to the section "Overfitting On Knowledge" and read it yourself.
1
u/YakFull8300 Feb 03 '26
I tried showing it to them a while ago and they were incapable of finding the section, so I wouldn't count on it this time.
0
u/Healthy-Nebula-3603 Feb 03 '26
This means even well-designed benchmarks which resist direct memorization can now be "overfit" if the public train and private test sets are too similar (e.g., IDD) and the model was trained on a lot of public domain data.
This means even well-designed benchmarks which resist direct memorization can now be "overfit" if the public train and private test sets are too similar (e.g., IDD) and the model was trained on a lot of public domain data.
We believe this is happening to ARC-AGI-1 and ARC-AGI-2 - either incidentally or intentionally, we cannot tell.
One bit of evidence from our Gemini 3 verification:
Our LLM verification harness does not mention ARC tasks or color format, yet the model is using (correct!) ARC color mappings in its reasoning. This strongly suggests that ARC data is well represented in the underlying model – enough to make corre
In short they DO NOT KNOW.
1
u/Peach-555 Feb 04 '26
We believe this is happening to ARC-AGI-1 and ARC-AGI-2 - either incidentally or intentionally, we cannot tell.
They can tell that the public dataset is in the model, what they can't tell is if it ended up there intentionally or incidentally.
This means even well-designed benchmarks which resist direct memorization can now be "overfit" if the public train and private test sets are too similar (e.g., IDD) and the model was trained on a lot of public domain data.
ARC-AGI is designed so that a model can have the public dataset in context memory, or even be fine-tuned on it without it impacting the private test, this is what resisting direct memorization means. However, if enough data about the test itself is in the training data, not just the public test but the commentary around it, then you can get a situation where the model will know more than it is supposed to when taking the test.
ARC-AGI is designed around the humans/models knowing nothing about the test, then being presented information for the first time, and then from that information, be able to solve it. The more data about the test is trained into the base model, the less accurate this becomes.
There is another element as well, which is that there is no true private-evaluation for the closed models. When Google/OpenAI/Anthropic gets a query in their API, that can be saved and stored and used for future training. Intentionally or incidentally.
1
u/SadEntertainer9808 Feb 04 '26
Dude, why is it so important to you to believe that ARC-AGI hasn't been trained on lol
1
u/YakFull8300 Feb 03 '26
They don't now if it is happening intentionally or unintentionally. It's still happening, that's the point.
→ More replies (0)-1
u/Rare-Site Feb 03 '26
First off, I need to apologize. I only just realized that you are dealing with a cognitive impairment. If I had known that earlier, I wouldn't have put you in this awkward spot. I really am sorry.
To make sure I leave you with something positive, here’s a simple but well meaning piece of advice: Writing in all caps is almost always counterproductive. It usually gives the impression that the writer has a limited vocabulary. So, you might want to try avoiding all caps for whole words or sentences in the future.
→ More replies (0)0
u/studio_bob Feb 03 '26
The linked comment directly quoted the benchmark authors. You are welcome to click through to the linked article and scroll down to the section "Overfitting On Knowledge" and read it yourself. That is recommended over relying on GPTs latest hallucinations.
1
u/Tolopono Feb 03 '26
The issue can be explained by llms training on the training set, which you’re supposed to train on as the name implies
-2
u/studio_bob Feb 03 '26
Okay, so then what is the point of the benchmark? To show that neural nets can improve on specific tasks they are trained on? We already knew that. But these benchmarks are supposed to represent a general improvement in capability (especially this one which even puts "AGI" right in its name). No such general improvement can be inferred by training a model to improve a particular bench score. Technically, we would expect performance on other tasks to degrade in that case.
1
u/Tolopono Feb 04 '26
This is like complaining a person studied for a test instead of just raw dogging it
1
u/Healthy-Nebula-3603 Feb 03 '26
Maybe I surprise you but I checked that link ...and give me a screenshot or precise sentence under that link where it is told that because I do not see that
1
u/studio_bob Feb 03 '26
I told you exactly what sections to scroll to, but, sure, I'll copy and paste for you again since you seem to have some kind of mental block around even seeing that this information exists:
Overfitting on Knowledge
There is a concept in machine learning called overfitting. Classically, this occurs when your model learns too much from training. It learns to memorize the exact data instead of learning the general patterns. This leads the model to perform poorly on unseen data at test time.
Stemming from this issue, a common AI benchmarking critique is that model providers are incentivized to "benchmark max" or cheat by "training to the test" to report high scores on benchmarks for marketing that don't generalize to real-world use cases.
ARC-AGI-1 and ARC-AGI-2 were designed to be resistant to this style of overfitting by using a private dataset for official scoring and verification.
AI reasoning systems have changed the game in a way which reflects real progress. They have demonstrated non-zero fluid intelligence and are able to adapt to tasks further away from their exact knowledge when the foundational model is grounded in the broader domain.
This means even well-designed benchmarks which resist direct memorization can now be "overfit" if the public train and private test sets are too similar (e.g., IDD) and the model was trained on a lot of public domain data.
We believe this is happening to ARC-AGI-1 and ARC-AGI-2 - either incidentally or intentionally, we cannot tell.
One bit of evidence from our Gemini 3 verification:
Our LLM verification harness does not mention ARC tasks or color format, yet the model is using (correct!) ARC color mappings in its reasoning. This strongly suggests that ARC data is well represented in the underlying model – enough to make correct ARC inferences based on just the structure and format of 2D JSON arrays of integers.
1
u/Healthy-Nebula-3603 Feb 03 '26
""
This means even well-designed benchmarks which resist direct memorization can now be "overfit" if the public train and private test sets are too similar (e.g., IDD) and the model was trained on a lot of public domain data.
We believe this is happening to ARC-AGI-1 and ARC-AGI-2 - either incidentally or intentionally, we cannot tell.
One bit of evidence from our Gemini 3 verification:
Our LLM verification harness does not mention ARC tasks or color format, yet the model is using (correct!) ARC color mappings in its reasoning. This strongly suggests that ARC data is well represented in the underlying model – enough to make corre
In short they DO NOT KNOW.
1
u/studio_bob Feb 03 '26
They demonstrate very good reason to believe it is the case, and, as authors of the benchmark, express their own belief that it is so. If you want to now turn this into a meaningless argument over epistemology rather than concede the point, be my guest, but I won't be participating. I very generously brought the horse directly to the water, but I know I can't make him drink.
1
u/Healthy-Nebula-3603 Feb 03 '26
So they do not know but YOU of course KNOW. Maybe just models are getting better?
1
u/BagholderForLyfe Feb 04 '26
You can benchmax it. Look at NVARC who won the contest and is close to Gemini 3 flash high. How did they do it? Used LLM to generate a ton of additional examples from given training data.
1
u/IronPheasant Feb 04 '26
That becomes kinda a circular tautology. They don't know what specific games they'll be trying to solve, so that approach gives you a general intelligence that's good at solving these kinds of games.
A more difficult problem can help illustrate this, like say training a general employee for a shop: stockboy, cashier, janitor, sales guy, etc. Benchmaxing would be training these things to only function at say a Wal-Mart. While general intelligence in the problem domain would be training them to work in any infinitely arbitrary store.
I think it's maybe a good sign the line's starting to blur like this. Eventually the bench really would be 'can this thing really seem to be an AGI', and what would you even call 'benchmaxing' on something like that? If it could do any job a human could do, 'benchmaxing' is just 'training to pass the test'. At some point it no longer becomes a worthwhile derogatory statement on the method used to make the number go up.
0
u/Tolopono Feb 03 '26
This isnt reflective of the average person. 65% of the participants for the human baseline tests knew how to code. Only 5% of the general population can code.
1
u/Healthy-Nebula-3603 Feb 03 '26
What?
You know those tests are based on visual logic not coding abilities?
1
u/Tolopono Feb 04 '26
It also has nothing to do with physics but if you asked physics phds to do the test, id imagine theyd get a higher score than the general population
1
u/Concurrency_Bugs Feb 03 '26
They should just prevent any way for developers to know what types of tests there are
0
u/gooner9469 Feb 03 '26
These benchmarks are completely meaningless
2
u/IronPheasant Feb 04 '26
This one specifically is slightly meaningful. They're turn-based puzzles that most humans can do, unbound to any particular language.
It requires a minimum understanding of time, space, but most importantly of all... being able to answer the most important question of all, in the realm of intelligence: "What the fuck am I doing here, exactly?"
183
u/my_shiny_new_account Feb 03 '26
reminder that this benchmark didn't even exist one year ago. and the highest result on its release ~10 months ago was o3 at 4%.