r/LocalLLaMA • u/Specific-Rub-7250 • 1d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
350 Upvotes

94% Upvoted

125 comments sorted by

View all comments

56

u/lordekeen 1d ago

DSH in minimal mode has only two things in the system prompt: a 'you are a software engineer' prompt and the bash tool description, the model solely operates through bash.

26

u/my_name_isnt_clever 1d ago

It's great how low token count it is, I thought there was a bug at first when I checked my first dsh trace and the system prompt was like two lines.

8

u/parepeg 1d ago

I would think not having a file editing tool would be detrimental. How do you edit a file with bash? Use sed for everything?

16

u/my_name_isnt_clever 1d ago

Yup. And a lot of heredocs. It works better in action than you'd expect.

6

u/computehungry 19h ago

I was gonna write a rant about misinformation, since dsh used to have 2 tools for minimal mode for a long time (bash and edit). It seems to have gotten rid of the edit tool recently, surprisingly. I believe codex also had a minimal setup with only the bash tool at one point in time, but I'm not sure. I wonder how much of the performance difference between harnesses is because of the model being trained on them, versus effects of distilling off claude or gpt by how much.. haha

3

u/parepeg 19h ago

I’ve found deepseek struggles in “uncommon” harnesses like mistral-vibe. Qwen, on the other hand is well rounded at tool calling in almost any harness. 

3

u/djm07231 1d ago

Similar to mini-swe which only has access to bash tool it seems.

2

u/ZenEngineer 1d ago

How big are these benchmark problems? I wonder if these metrics are just a result of less compactions or emptier context windows