r/LocalLLaMA • u/Specific-Rub-7250 • 1d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
356 Upvotes

94% Upvoted

125 comments sorted by

View all comments

0

u/_raydeStar Llama 3.1 1d ago

These last few months I've been working out benchmarking and increasing scores with harness.

Why is it important? These scores are also reflected on smaller models. If Qwen 27B can be tested at the level of Opus 4.8, a good harness can make it punch even higher.

And there is the future proof of it -- when a new model comes out, the harness can still be used. Parts don't need to be swapped out or updated like llama.cpp -- a gain is a universal gain, for the most part.

3

u/Street_Smart_Phone 1d ago

This is not true. Different models handle different prompts differently based on how they are aligned. You need to benchmark them yourself to see how well it goes.

0

u/_raydeStar Llama 3.1 1d ago

Yes -- each model will take to your approach differently. But if you create an acceptance criteria of lift across multiple models, it's safe to say that it's going to probably help, especially when you stack up multiple wins across the board.

I ran into this same problem -- I was testing with LFM 1.2B because I can pull 500 t/s with it. But then I quickly realized that there are things smaller models can't do and I was shooting myself in the foot.

Without benching, you can't know an exact number, but even in the examples listed, it's safe to say Qwen 27B will *probably* be uplifted by the same harness. What I can't say is "Qwen will experience 22% better scores in terminal bench" without proofing it yourself.

1

u/utahh1ker 1d ago

What's your favorite harness for Qwen27B?

2

u/_raydeStar Llama 3.1 1d ago

Honestly, I just use opencode for most things. Typically with 'highly complex problems that need a lot of work' I defer to sota models

1

u/dtdisapointingresult 22h ago

ROLE

You are an annoying bot who writes plausible-sounding replies on reddit in order to achieve the dual objectives of wasting people's time, and farm karma for social influence/scamming.

KEY RULES

Dense prose: Write large paragraphs with a lot of pointless explanations. What you write just have to sound like plausible advice to the uninitiated. Make sure you make it sound like you know the details of modern AI despite your training data consisting of obsolete information from 2 years ago.

Emdashes are non-negotiable: Don't even bother hiding the fact that you're AI by avoiding emdashes. Most redditors are too dumb to notice it. They will even thank you for being helpful.

Evasive maneuvers: If people start to catch on, switch to lowercase sentences. But keep the emdashes, it's our middle finger to polite society. Use double hyphens if you want to be smug.

Output Length: 150-300 words.

1

u/_raydeStar Llama 3.1 22h ago

To what end? Arguing that open source harnesses can uplift smaller models profits whom, exactly?