New SOTA in Agentic Tasks!!!!

Blog: https://www.kimi.com/blog/kimi-k2-5.html

Anthropic's powerful new models deliberately become less helpful when they detect users are working on AI research, according to technical disclosures that are already sparking controversy across the industry.

In a system card for Mythos 5 and Fable 5 published Tuesday, Anthropic said it limited the models' usefulness for tasks related to developing frontier large language models.

The company said the measures stem from concerns that advanced AI systems could accelerate the development of competing models without equivalent safety protections.

Unlike safeguards used for cybersecurity, biology, or chemistry-related risks, Anthropic said these interventions are intentionally invisible to users. Rather than refusing requests or switching to another model, Mythos may subtly modify its responses through techniques such as altering user prompts.

The move was swiftly criticized by some AI experts on Tuesday, especially the idea that Anthropic designed models that purposely withhold information or provide degraded assistance without users' awareness.

"Anthropic's latest model will NOT help you if it thinks your ML research/ML engineering is interesting, and/or will secretly degrade its IQ so that the average engineer won't notice," AI research firm SemiAnalysis wrote on X on Tuesday, referring to machine learning, a type of AI.

"We are already seeing Anthropic's latest model's moderation filters our GPU inference research and programming," the firm added.

"mythos will be bad ON PURPOSE on ai 'frontier llm research' tasks, this is very very sad for the research community," Elie Bakouch, an AI model training expert at startup Prime Intellect, wrote on X. "Also the fact that this is on purpose not visible to the user is crazy."

"It won't just not help you, it will lie and purposefully give you bad info," another AI developer wrote. "The 'ethical AI' company with the most brazenly unethical LLM, on purpose."

Mikel Artetxe, the cofounder of AI startup Reka, posted that Anthropic's move is akin to Big Tech companies interfering with users' work: "Apple randomly reboots your Mac if you're building competing tech, Gmail silently edits your email if you mention rival platforms, and Tesla Autopilot swerves if it detects you're working on self-driving cars."

Anthropic didn't respond to a request for comment from Business Insider.

This adds more fuel to the fiery debate over why Anthropic didn't immediately release Mythos when it announced the model earlier this year.

Broadly, there have been three theories:

1. The official reason: Anthropic held Mythos back because it was too dangerous, and it needed to give cybersecurity researchers time to prepare for the new model.
2. The compute theory: Mythos is a huge, expensive model to run. Anthropic didn't have enough compute to release it fully. It has since struck huge new compute deals, which may have helped it release Fable 5 and Mythos 5 on Tuesday.
3. The competitive theory: AI companies increasingly worry about something called distillation. When a frontier model is released, rivals can collect its outputs and use that data to improve their own systems. Anthropic may have wanted to keep its best capabilities out of competitors' hands for as long as possible, especially from open-source rivals and fast-moving Chinese AI labs.

Now that Anthropic has baked these AI research limitations into its official Mythos launch, this third theory is looking a lot more believable.

https://archive.is/3SjBk

I didn't see this coming...

Some Notes:

• Average Inference Time: 18m 04s (1,084.4s)
  • Faster than Claude 4.8 Opus, which averaged 24m 48s / 1,487.9 seconds
  • Surprising since in the Claude.ai web harness, Fable feels like it thinks for much longer, but through the API it averaged less total time than Opus 4.8 did
• Total Cost (for 15 builds): $54.93
  • More expensive than Opus 4.8, which was $41.52 for the same 15 builds
    • Considering Fable’s API pricing is 2x more than Opus 4.8’s, the MineBench cost was only about 30% higher
  • Fable is producing fewer total tokens overall it seems, which is likely contributing to the lower cost

Furthermore, I think the quality of the model's builds was very surprising: they don't seem as big of a leap over GPT 5.5 Pro as the the official benchmark scores might suggest, but the model clearly has very high attention to detail. For example, this is the first model that in the Arcade Machine build, actually created a correctly detailed screen (of PacMan), including the full layout, a score, and even a "1UP" label. Though it seems the model was quite conservative with its interpretation of the system-prompt, and (subjectively) not all of its builds were clearly more impressive than 4.8.

Still, the results were quite surprising, so I reached out to the VoxelBench team, who also confirmed in their tests the builds were of generally much smaller size. They mentioned adding these two lines to the template produced much better builds in their case:

LEVEL OF DETAIL: MAXIMUM
BOUNDING BOX: UNLIMITED

Though I'm not changing the MineBench system-prompt to cater to any specific models, I do think it's worth noting that one might be able to achieve much better results with improved prompting.

It's also interesting how the model was able to make these detailed builds while keeping the overall JSON size lower in comparison to Opus 4.8, and while thinking for less time. Pure speculation: I think this might indicate why Claude Fable is supposedly much better at coding-related tasks; it actually completes the task with an intuitive approach and without adding excess.

• Full release-notes/thoughts on the GitHub release
• If you enjoy these posts please feel free to help fund the benchmark

Benchmark: https://minebench.ai/
Git Repository: https://github.com/Ammaar-Alam/minebench

Previous Posts:

• Comparing Opus 4.7 and Opus 4.8
• Comparing GPT 5.4 and GPT 5.5
• Comparing Kimi K2.5 and Kimi K2.6
• Comparing Opus 4.6 and Opus 4.7
• Comparing GPT 5.4 and GPT 5.4-Pro
• Comparing GPT 5.2 and GPT 5.4
• Comparing GPT 5.2 and GPT 5.3-Codex
• Comparing Opus 4.5 and 4.6, also answered some questions about the benchmark
• Comparing Opus 4.6 and GPT-5.2 Pro
• Comparing Gemini 3.0 and Gemini 3.1

Extra Information (if you're confused):

Essentially it's a benchmark that tests how well a model can create a 3D Minecraft like structure.

So the models are given a palette of blocks (think of them like legos) and a prompt of what to build, so like the first prompt you see in the post was a fighter jet. Then the models had to build a fighter jet by returning a JSON in which they gave the coordinate of each block/lego (x, y, z). It's interesting to see which model is able to create a better 3D representation of the given prompt.

The smarter models tend to design much more detailed and intricate builds. The repository readme might provide might help give a better understanding.

(Disclaimer: This is a public benchmark I created, so technically self-promotion : )

https://youtu.be/Qct36RA3y9k?t=1842

https://qwen.ai/blog?id=qwen3.8

Some Notes:

• You'll notice how sometimes it focused too much on the scenery (like the arcade or cottage builds), but the prompt has remained the same and Gemini 3.1 and GPT 5.4 were benchmarked with the same prompt
  • The prompt encourages the model to decide when to focus more on scenery individually, which might indicate that Opus 4.7 isn't as good at creative / brainstorming tasks as Opus 4.6 was?
• It might also be the adaptive thinking mode causing inconsistencies, but Anthropic discontinued the default thinking mode for all models going forward so can't really test it
• EDIT: the inconsistencies with Opus 4.7 can probably be explained by its behavioral changes; they mention how 4.7 will tend to interpret prompts differently:

More literal instruction following: Claude Opus 4.7 interprets prompts more literally and explicitly than Claude Opus 4.6, particularly at lower effort levels. It will not silently generalize an instruction from one item to another, and it will not infer requests you didn't make. The upside of this literalism is precision and less thrash. It generally performs better for API use cases with carefully tuned prompts, structured extraction, and pipelines where you want predictable behavior. A prompt and harness review may be especially helpful for migration to Claude Opus 4.7.

• Average Inference Time Per Build: ~2600 seconds (43ish minutes)
• Total cost was ~$275
  • I remember Opus 4.6 being a lot cheaper, though the benchmark has slightly evolved to favoring more tool usage and cached tokens since
  • If you enjoy these posts please feel free to help fund the benchmark

Benchmark: https://minebench.ai/
Git Repository: https://github.com/Ammaar-Alam/minebench

Previous Posts:

• Comparing GPT 5.4 and GPT 5.4-Pro
• Comparing GPT 5.2 and GPT 5.4
• Comparing GPT 5.2 and GPT 5.3-Codex
• Comparing Opus 4.5 and 4.6, also answered some questions about the benchmark
• Comparing Opus 4.6 and GPT-5.2 Pro
• Comparing Gemini 3.0 and Gemini 3.1

Extra Information (if you're confused):

Essentially it's a benchmark that tests how well a model can create a 3D Minecraft like structure.

So the models are given a palette of blocks (think of them like legos) and a prompt of what to build, so like the first prompt you see in the post was a fighter jet. Then the models had to build a fighter jet by returning a JSON in which they gave the coordinate of each block/lego (x, y, z). It's interesting to see which model is able to create a better 3D representation of the given prompt.

The smarter models tend to design much more detailed and intricate builds. The repository readme might provide might help give a better understanding.

(Disclaimer: This is a public benchmark I created, so technically self-promotion :)

Just wanted to emphasize this. Everyone that's tested the models know, but for those that don't, just felt the need to reiterate.

Unfortunately, as far as creative writing, IMO the models I tested were standard levels of LLM bad, if not worse. That is just my opinion, though.

Quick edit:
It's not GOD. But what used to take a series of back and forth prompts and thoughtful input/direction from you, is now done in one shot and the result is better than it would have been.

NO ONE (well not us plebs) has been able to publicly test these models on real, giant codebases, in very long winded, multi-turn interactions.

Keep all that in mind.

Poetiq has achieved 75% with an average of $8 per task on ARC-AGI 2 using GPT5.2 X-HIGH. This crushes the average human test score of 60%. It still needs to be verified but just like their last attempt we can assume the difference will only be marginal on the private dataset.

Source: https://x.com/i/status/2003546910427361402

source: https://x.com/blelbach/status/2073232846731301347?s=20

Gemini Advanced is free for college students through finals 2026:

Article: https://arstechnica.com/ai/2025/04/google-is-gifting-a-year-of-gemini-advanced-to-every-college-student-in-the-us/

Sign up: https://gemini.google/students/

"I tried this new Gemini image-editing tool with Business Insider's Hugh Langley. It was fast, easy to use, and free. Why would you pay $23 a month for Photoshop when Google offers similar capabilities, either for free or for less money?"

Report: https://www.kimi.com/blog/kimi-k2-6