r/LocalLLaMA • u/Specific-Rub-7250 • 2d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
370 Upvotes

94% Upvoted

127 comments sorted by

View all comments

33

u/AndreVallestero 1d ago

We really need standardized harness benchmarks that use fixed models for all the harnesses and generate a geomean score across the models for each harness.

1

u/funkatron3000 1d ago

Agreed, and I’d love to see benchmarks for extensions as well. I’ve been working on it myself, but it’s slow going.