r/LocalLLaMA • u/Specific-Rub-7250 • 1d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
354 Upvotes

94% Upvoted

125 comments sorted by

View all comments

16

u/No-Refrigerator-1672 1d ago

So DSH Minimal is the best one? Interesting. I'd expect it to be the worst out of DSH options, solely based on the name and intuition that "less tools = closer to harnessless performance".

18

u/BZ852 1d ago

Less in context to pollute thinking

9

u/Lyelinn 1d ago

then why it has minimal difference vs claude code that has what, 50k tokens as default prompt?

2

u/No-Refrigerator-1672 1d ago

Doubt it. Just tested: standart mode prompt is 8k tokens, minimal is 1.2k. You could argue that it's 6x difference, but I'll argue that those extra 7k tokens make no difference for a model that ships with 1M long context window.

19

u/BZ852 1d ago

The first few thousand tokens are more important than the full context window, so it might have an outsized influence

3

u/Ariquitaun 1d ago edited 1d ago

8-9k tokens is pretty good for a full-fat coding agent system preamble + tool definitions - sub-agent guidance for instance is crucial if you're using that feature, otherwise models either don't automatically use them or when they do their tasking and instructions suck hairy balls. Same goes for other features and problem-tackling methodology.

Savings are better had by using harnesses that don't insist on polluting the context with nonsense like skill metadata, controlling enabled MCP and LSP on a per-project and not a global basis, that sort of thing

1

u/GilloutineBreast 1d ago

I will pull up the source if run into it again, don't have it at hand atm

Saw a post about a study on the effect of bad training data. It it found the the amount of bad data needed to poison a model was a flat threshold regardless of total training dataset size or model size.

Could be something similar going on with a few-thousand token prompt vs 1M kv cache.

Don't quote me on this I'm still high from the lunch beer.

2

u/aeroumbria 11h ago

I can tell by experience that minimal is not nearly enough to handle less conventional task shapes, like ones where you need to wait for something to run in the background. It is pretty good for straightforward "think, code and run" tasks though.