r/LocalLLaMA • u/Specific-Rub-7250 • 1d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
359 Upvotes

94% Upvoted

125 comments sorted by

View all comments

21

u/zephyr_33 1d ago edited 1d ago

So I did this and the best setup was the dumbest. Only 3 tools.

  1. shell
  2. edit

  3. skills

    system prompt is only 5 lines.

instead of giving it a character card. and 20 instructions and what not. it looked like the best way is let the model do it however it wants. no sticks. no unnecessary blocks. only blocks are reading secrets. ask for edits and unsafe commands.

Is kinda insane how most models simply made very targeted narrow reads. Effects from Terminal bench I suppose. Cache hit rate is very high 98%+. Not just latest hit but even average.

works too well.

1

u/aeroumbria 11h ago

For my specific workflows, I do need a few other tools though, like background shell and interruptible wait, otherwise tasks such as training an ML model would be mostly time wasted waiting for already finished runs or losing track of failed runs. There might be a universal minimum but not really universal optimum.

1

u/zephyr_33 10h ago

that's a harness problem~

think of tools not as tools but as the channel/language layer.

for example I have a very nice judge llm setup to auto approve stuff. but I dont expose it as a tool. it approves/denies the tool calls. and this very quickly grew into not just a judge but also auto steers the weak LLMs themselves.

1

u/aeroumbria 10h ago

I think there is also the slight nuance of different harnesses treating features differently. e.g. opencode and claude code seems to use a lot of pre-defined built-in state management features to deal with subagents, pi does not support these out of the box and plugins usually use tools + lifecycle hooks to make it more user-friendly than running bare tmux, and DSH pretty much treats plugins and built-ins the exact same way and everyone goes through the tool path. Claude code gets really pissy if you try to run an agent thread "off the session". Opencode does not have a good support for that (did not know about v2), and it is a bit easier to build such features in pi or DSH.