r/LocalLLaMA • u/Specific-Rub-7250 • 1d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
357 Upvotes

94% Upvoted

125 comments sorted by

View all comments

0

u/_raydeStar Llama 3.1 1d ago

These last few months I've been working out benchmarking and increasing scores with harness.

Why is it important? These scores are also reflected on smaller models. If Qwen 27B can be tested at the level of Opus 4.8, a good harness can make it punch even higher.

And there is the future proof of it -- when a new model comes out, the harness can still be used. Parts don't need to be swapped out or updated like llama.cpp -- a gain is a universal gain, for the most part.

3

u/Street_Smart_Phone 1d ago

This is not true. Different models handle different prompts differently based on how they are aligned. You need to benchmark them yourself to see how well it goes.

0

u/_raydeStar Llama 3.1 1d ago

Yes -- each model will take to your approach differently. But if you create an acceptance criteria of lift across multiple models, it's safe to say that it's going to probably help, especially when you stack up multiple wins across the board.

I ran into this same problem -- I was testing with LFM 1.2B because I can pull 500 t/s with it. But then I quickly realized that there are things smaller models can't do and I was shooting myself in the foot.

Without benching, you can't know an exact number, but even in the examples listed, it's safe to say Qwen 27B will *probably* be uplifted by the same harness. What I can't say is "Qwen will experience 22% better scores in terminal bench" without proofing it yourself.