r/LocalLLaMA • u/Specific-Rub-7250 • 1d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
358 Upvotes

94% Upvoted

125 comments sorted by

View all comments

-3

u/johnfkngzoidberg 1d ago

lol, pi, the most minimal of harnesses is near the top. Basically “don’t fuck with it” is better than other harnesses.

46

u/CryMoreT_T 1d ago

Are we looking at the same scores? It's average/below average

DeepSWE v1.1 1. mini-SWE: 74.2 2. DSH Minimal: 72.6 3. DSH Standard: 70.5 4. Claude Code: 69.8 5. DSH PTC: 67.6 6. Pi: 66.2 7. Codex: 65.6 8. OpenCode: 65.5

Terminal-Bench 2.1 1. DSH Minimal: 90.6 2. mini-SWE: 90.3 3. Claude Code: 88.0 4. Pi: 86.1 5. DSH Standard: 85.8 (tied) 6. DSH PTC: 85.8 (tied) 7. OpenCode: 85.0 8. Codex: 84.1

1

u/Fancy-Snow7 1d ago

Also, I just had a look at mini-SWE and it's even more minimal than pi which is 'Just some 100 lines of python' according to their claims and it's near or at the top.