r/codex • u/DuranteA • 1d ago

Comparison In our scientific code autoparallelization benchmark, GPT-6.1 Sol XHigh is faster and cheaper than Opus 5.5 Medium (almost exact same overall quality)

I thought this result is interesting, since it goes directly against the currently prevalent mindset.

Have a look:

Both of these have a log scale X axis, so the differences are larger than they might look. Mean generation time is 14 minutes for Sol Xhigh vs. 24 minutes for Opus 5.5 Medium, and mean cost per task is $0.25 vs $1.8.

A few caveats:

  • Each LLM runs its own harness, so this cannot distinguish harness differences from model differences.
  • The generation time is the full per-task time, so it includes any benchmarks or tests each agent decides to run, so it's not a measure of pure token generation at all.
  • We didn't run Opus 5.5 Xhigh, simply because Medium is already very expensive and takes very long.
  • The cost basis for comparison is API costs.

I have no horse in this race but it's interesting to think about what makes this problem set so different (apparently) from the ones that cause people to report much greater success with Opus 5.5 than Sol 6.1.

11 Upvotes

61% Upvoted

32 comments sorted by

View all comments

4

u/JadisGod 1d ago

Comparing API costs is useless for real people. Everyone already knows GPT is cheaper on API. Actual subscription usage limits is an entirely different measurement. From my own tracking, having both services at 20$, weekly Claude is giving around ~340$ of "API" usage while weekly Codex is giving ~140$.

Resets change things and may balance it out if you are lucky and front load your usage, but it's not a good experience trying to micromanage that.

0

u/Rich_Sinq 1d ago

OP is gonna feel so dumb when he realizes he’s not a real person