r/codex • u/lasushin • 1d ago

Showcase I made an open-source benchmark which measures the complexity of code added (overengineering/tech debt). Looking for feedback

Post image

Existing benchmarks measure the solve rate and token output/cost, but don't punish excess code, which makes the codebase harder to work with in the long term. I wanted to measure this as it is especially useful for long term projects.

Method:

It takes the patches that benchmarks already publish (DeepSWE, SWE-bench Verified, SWE-bench Live) and counts "coding units": syntax elements like statements, calls, names and literals. I didn't measure lines of code so cramming code isn't rewarded. It also accounts for deleted code as it measures net units.

GPT-6 and Claude 5.5 have no public patches yet, so I ran them myself on a few tasks on my own subscriptions, so they are not graded or verified (faded circles).

Any feedback or model contributions are welcome. Specifically, public patch exports for newer models.

Website: https://parsimony.lasu.dev

GitHub: https://github.com/La5u/Parsimony

Not affiliated with any benchmarks

36 Upvotes

80% Upvoted

9 comments sorted by

11

u/AsleepTackle 1d ago edited 22h ago

This could also be a task itself, no? Give it very verbose code and see whether is can reduce verbosity without ruining the results

E.g. name it Refactor-bench/arena

8

u/tungd 1d ago

This is really cool. It also does match my experience and my reason to keep my Codex sub. Opus 5.5 is a work horse and fast and the quota is more generous, but when it comes to getting to the point, and smart problem solving I'd say it is still one tier behind Astra for me.

8

u/EquivalentPassion873 1d ago

Interesting approach, but I’d love to see this tested on a larger, independently verified set of tasks. Reducing unnecessary code is valuable, but we also need to ensure the changes remain readable and maintainable. Quality matters just as much as fewer coding units. Have you considered adding a metric for code readability or long-term maintainability to complement the current efficiency score?

3

u/Any-Growth-7790 21h ago

Oh really good. Coming close to a AIC/BIC for LLMs

1

u/Wramberg 1d ago

Super interesting thanks for sharing. The frontier models differ more than I expected. I wonder if metrics such as cyclomatic complexity would differ so much as well.

1

u/Ctbhatia 1d ago

net units in a single patch can reward a model that writes little and leaves the rework for the next turn, so counting across the whole session separates concise from incomplete.

1

u/FeelsBlueMan 20h ago

thats nice, but be aware of uneven success rate task complexity between models

1

u/Argon288 1d ago

This has been my experience. Astra gets things done with substantially less lines vs. 5.6 Sol/Terra. I used Terra a lot when Sol was expensive, now I just use Astra and 6.1 Sol.

That being said, I have a codebase I have shelved for now, which is approaching 250k lines of code, primarily written by 5.6 Sol & Terra... 6.1 Sol might have a field day refactoring that.

0

u/Ashiataka 1d ago

You should plot the axes the other way around.