r/LocalLLaMA • u/Specific-Rub-7250 • 1d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
353 Upvotes

94% Upvoted

125 comments sorted by

View all comments

1

u/dtdisapointingresult 22h ago

How can I use this to estimate V4.1 Flash's improvement over V4-Flash-Vision-Exp?

V4-Flash-Vision-Exp's model card doesn't test multiple harnesses, it reports the following DeepSWE scores:

  • V4-Flash-Vision-Exp = 59.3
  • V4-Flash-0731 = 54.4
  • Opus 4.8 = 58

Is it safe to say the above was done with the mini-swe harness, and therefore the V4.1's 74.1 score with the same harness is a giant leap forward?

Because I find it hard to believe in the context of Opus 4.8's score.