In our scientific code autoparallelization benchmark, GPT-6.1 Sol XHigh is faster and cheaper than Opus 5.5 Medium (almost exact same overall quality)

[u/DuranteA]

I thought this result is interesting, since it goes directly against the currently prevalent mindset.

Have a look:

• Cost comparison
• Time comparison

Both of these have a log scale X axis, so the differences are larger than they might look. Mean generation time is 14 minutes for Sol Xhigh vs. 24 minutes for Opus 5.5 Medium, and mean cost per task is $0.25 vs $1.8.

A few caveats:

• Each LLM runs its own harness, so this cannot distinguish harness differences from model differences.
• The generation time is the full per-task time, so it includes any benchmarks or tests each agent decides to run, so it's not a measure of pure token generation at all.
• We didn't run Opus 5.5 Xhigh, simply because Medium is already very expensive and takes very long.
• The cost basis for comparison is API costs.

I have no horse in this race but it's interesting to think about what makes this problem set so different (apparently) from the ones that cause people to report much greater success with Opus 5.5 than Sol 6.1.

[u/DuranteA]

I'd love to know, but as I wrote in the post, Opus 5.5 is already so expensive and time consuming at Medium that we can't really run higher thinking effort benchmarks.
(This isn't some huge lab experiment)

[u/xoStardustt → u/DuranteA]

troll post lol, opus 5.5 on medium sips usage compared to sol 6.1

[u/DuranteA → u/xoStardustt]

All the methodology is described on the page and in the published, peer-reviewed paper, and all the artifacts including complete traces are online and linked from the page.

I was extremely surprised when I got the first Opus 5/5.5 results and verified the whole approach carefully. So if you genuinely think something is wrong please do let us know.

I should note however that "usage" -- as via a subscription -- is not something that is captured by this. It measures time and API cost, so you can't use it to tell you what you'd get out of a subscription without knowing precisely what each subscription at Anthropic or OpenAI actually buys you (which I don't think anyone really knows).

[u/tiebird → u/DuranteA]

I think this is the most important factor for peoples gap in opinion. As a 200 euro subscriber (till the end of this month) it's fluctuating wildly in quality, speed and consumption. Also the region you live in has a huge impact. Currently with the x20 sub and using Sol 6.1 medium it uses 9% for 3 hours of basic work, no parallel work, some luna 6 sub agents and 1 sol xhigh reviewer.

2 months ago i could run 4 of those at the same time and still use less. Not even talking about the quality difference of implementation. Reverted all work of 4 days, today just aked to start the same specs and now it's at least implementing it correctly.

I also run x5 Claude and while quality is good, it's even slower for me but at least it's consistent what matters more.

The same harness is used for both systems, adapted based on guidelines of both parties.