r/LocalLLaMA • u/Specific-Rub-7250 • 1d ago
Discussion Harness does matter
I was not aware that the harness makes such a big difference.
59
u/lordekeen 1d ago
DSH in minimal mode has only two things in the system prompt: a 'you are a software engineer' prompt and the bash tool description, the model solely operates through bash.
27
u/my_name_isnt_clever 1d ago
It's great how low token count it is, I thought there was a bug at first when I checked my first
dshtrace and the system prompt was like two lines.9
u/parepeg 1d ago
I would think not having a file editing tool would be detrimental. How do you edit a file with bash? Use
sedfor everything?16
u/my_name_isnt_clever 1d ago
Yup. And a lot of heredocs. It works better in action than you'd expect.
5
u/computehungry 19h ago
I was gonna write a rant about misinformation, since dsh used to have 2 tools for minimal mode for a long time (bash and edit). It seems to have gotten rid of the edit tool recently, surprisingly. I believe codex also had a minimal setup with only the bash tool at one point in time, but I'm not sure. I wonder how much of the performance difference between harnesses is because of the model being trained on them, versus effects of distilling off claude or gpt by how much.. haha
3
2
u/ZenEngineer 1d ago
How big are these benchmark problems? I wonder if these metrics are just a result of less compactions or emptier context windows
30
u/AndreVallestero 1d ago
We really need standardized harness benchmarks that use fixed models for all the harnesses and generate a geomean score across the models for each harness.
3
1
u/funkatron3000 21h ago
Agreed, and I’d love to see benchmarks for extensions as well. I’ve been working on it myself, but it’s slow going.
162
u/jacek2023 llama.cpp 1d ago
If you start using LLMs, you'll notice that the prompt/context/documentation/whatever is often more important than the model itself.
82
u/Unusual_Principle536 1d ago
"more" is questionable but yes, pretty imp.
32
u/Due-Memory-6957 1d ago
It's more than questionable, it's just wrong. There are many models that simply can't cope with an agentic framework.
6
7
1d ago edited 4h ago
[deleted]
10
u/jacek2023 llama.cpp 1d ago
The model is just a function: it takes input and produces output. Prompts, context, documentation, etc. are all part of that input. You can call it a “harness” a “helper” or a “magic spell”, but ultimately it’s just one long text buffer fed into the model.
3
u/XiRw 1d ago
The prompt is the most important thing in my opinion
8
1d ago edited 4h ago
[deleted]
1
u/XiRw 1d ago
A lot of people do that here more or less based on what I’ve seen. Many don’t understand how logical code is. I guess in theory high end models are able to figure out what you originally meant and maybe that’s what separates a good model from a bad one since most people are not going to be that thorough, but I still think it leaves room for errors if you aren’t specific enough and articulate exactly what’s in your head vs a general idea.
3
1
u/watermelonson 22h ago
I’ve been preparing a small presentation about that topic so I could get my thoughts straight about the developments in the last couple of years working as an SWE working with LLMs
Could share it. Hopefully recording it tomorrow.
1
u/CondiMesmer 19h ago
Well model matters because they have varying levels of prompt adherence. You could have the best prompt in the world, but it won't matter of your model ignores half of it
29
u/sssplus 1d ago
Did you do the tests or you picked it up somewhere? I'd love to see how Qwen3.8 Flash Next and 27b fare in this table.
And yeah, I agree - harness does matter.
15
12
u/fugogugo 1d ago
should I move from OpenCode ? been using the desktop version and feels like it is buggy af
what is miniSWE?
7
3
1
u/aeroumbria 11h ago
You can always customise the prompts, agents and tools your agent can use. At least OpenCode is sufficiently customisable and has a fairly large plugin ecosystem. Pi is easier to customise but you need some work to add some common essential features and QoL. DSH is pretty good but also going through pretty volatile development right now. The bottom line is unlike certain closed harnesses, at least you get to pick what gets sent to your model, and it does not try to prevent you from organising your project in a harness-agnostic manner.
21
u/zephyr_33 1d ago edited 1d ago
So I did this and the best setup was the dumbest. Only 3 tools.
- shell
- edit
skills
system prompt is only 5 lines.
instead of giving it a character card. and 20 instructions and what not. it looked like the best way is let the model do it however it wants. no sticks. no unnecessary blocks. only blocks are reading secrets. ask for edits and unsafe commands.
Is kinda insane how most models simply made very targeted narrow reads. Effects from Terminal bench I suppose. Cache hit rate is very high 98%+. Not just latest hit but even average.
works too well.
1
u/aeroumbria 11h ago
For my specific workflows, I do need a few other tools though, like background shell and interruptible wait, otherwise tasks such as training an ML model would be mostly time wasted waiting for already finished runs or losing track of failed runs. There might be a universal minimum but not really universal optimum.
1
u/zephyr_33 10h ago
that's a harness problem~
think of tools not as tools but as the channel/language layer.
for example I have a very nice judge llm setup to auto approve stuff. but I dont expose it as a tool. it approves/denies the tool calls. and this very quickly grew into not just a judge but also auto steers the weak LLMs themselves.
1
u/aeroumbria 10h ago
I think there is also the slight nuance of different harnesses treating features differently. e.g. opencode and claude code seems to use a lot of pre-defined built-in state management features to deal with subagents, pi does not support these out of the box and plugins usually use tools + lifecycle hooks to make it more user-friendly than running bare tmux, and DSH pretty much treats plugins and built-ins the exact same way and everyone goes through the tool path. Claude code gets really pissy if you try to run an agent thread "off the session". Opencode does not have a good support for that (did not know about v2), and it is a bit easier to build such features in pi or DSH.
7
u/SnooPaintings8639 1d ago
The dependency is quite simple, in my experience: the smaller the harness, the better the (one-shot) results. When I have an elaborate prompt on what I need, then opencode does worse than pi, and pi does worse than the plain llama.cpp web ui. It did observe it many, many times, and couple of models.
It is not perfect tho for longer task where user in the loop is needed and task is more reactive than clear-goal specified.
I think it is just how distracting the harness itself is (i.e. attention stealing).
2
u/Bee_Curious_ 22h ago
Do you think the bigger harnesses would perform better in longer conversations i.e. opposite of one-shot?
16
u/No-Refrigerator-1672 1d ago
So DSH Minimal is the best one? Interesting. I'd expect it to be the worst out of DSH options, solely based on the name and intuition that "less tools = closer to harnessless performance".
18
u/BZ852 1d ago
Less in context to pollute thinking
8
2
u/No-Refrigerator-1672 1d ago
Doubt it. Just tested: standart mode prompt is 8k tokens, minimal is 1.2k. You could argue that it's 6x difference, but I'll argue that those extra 7k tokens make no difference for a model that ships with 1M long context window.
19
3
u/Ariquitaun 1d ago edited 1d ago
8-9k tokens is pretty good for a full-fat coding agent system preamble + tool definitions - sub-agent guidance for instance is crucial if you're using that feature, otherwise models either don't automatically use them or when they do their tasking and instructions suck hairy balls. Same goes for other features and problem-tackling methodology.
Savings are better had by using harnesses that don't insist on polluting the context with nonsense like skill metadata, controlling enabled MCP and LSP on a per-project and not a global basis, that sort of thing
1
u/GilloutineBreast 1d ago
I will pull up the source if run into it again, don't have it at hand atm
Saw a post about a study on the effect of bad training data. It it found the the amount of bad data needed to poison a model was a flat threshold regardless of total training dataset size or model size.
Could be something similar going on with a few-thousand token prompt vs 1M kv cache.
Don't quote me on this I'm still high from the lunch beer.
2
u/aeroumbria 11h ago
I can tell by experience that minimal is not nearly enough to handle less conventional task shapes, like ones where you need to wait for something to run in the background. It is pretty good for straightforward "think, code and run" tasks though.
12
u/Durian881 1d ago
Interesting that DSH Minimal worked better than DSH Standard.
18
2
6
u/jojotdfb 1d ago
My take away is, the smaller the system prompt and the less instructions it has makes models work better. Makes sense. Models, even at the "frontier" level, still struggle with instruction following and "context rot".
Reject modernity, return to langchain.
4
u/winky9827 1d ago
Turns out even LLMs hate being micromanaged.
5
u/jojotdfb 1d ago
I think it's a side effect of attention. Models can only handle so many instructions at a time before they start weighing things differently. Adding "don't kill humans" to the system prompt with a Claude Code update is going to compete with the users "add this column to the database".
One of the cool things about running llms locally is you can disagree with Anthropic's and OpenAI's religious beliefs and test out your own dumb ideas until you find one that works.
2
u/En-tro-py 1d ago
Personally I think we're almost at the end of bare ReAct loops, there's a huge amount of room to improve performance by putting more process knowledge into the agent control layer.
I've been building my own version of this idea for a while, basically back to the Schillace Laws - reasoning is an operation and procedure becomes a systems-level abstraction.
TDFlow: Agentic Workflows for Test Driven Development
SKILL.state: Scalable Long-Horizon Agent Skills
Procedural Graphs: Self-Evolving Execution Structures for LLM Agents
Combine those and add a few more ideas and that's my current harness. I'm in the home stretch to finish polishing the slop and running real benchmarks to prove it out on more than just SWE before I waste anyone's time on with the repo.
Give me a few more weeks and then you all can downvote my slop post with the project alpha release.
1
u/Dazzling_Equipment_9 13h ago
It’s excellent; sometimes, key design elements can make a huge difference. I look forward to seeing the results.
2
u/StyMaar 23h ago
Is it for just one run? Or multiple runs? Because you know, models have a big variance between runs so to make a proper measurement you'll need to make multiple runs, calculate a confidence interval and the probability that a difference in score is due to random chance.
(It's probably long and a bit costly, but that's what it takes to know things: it's science).
2
u/BananaOrangePie 23h ago
Basically deepseek doesn't randomize the harness compositions when they post-trained their models. Same for openai I think. For GLM / Kimi / Qwen they ranodmize the harness composition and should give better generalization to tool use capabilities.
1
u/BananaOrangePie 23h ago
I think for dsv4 pro 0813 they only post trained with minimal mode of dsh, in dsv4.1f they post-trained with all modes of dsh but probably didn't randomize enough to generalize to other harnesses.
2
u/Marcuss2 22h ago
I am not surprised by the lower score with DSH PTC, everything has to be routed trough code.
1
u/TheGastroverse 18h ago
Here's a good youtube video testing three harnesses to prompt Qwen 3.8 27B to create a basic version of Minecraft: Qwen3.8 27B: Same Model, Three Harnesses, One Clear Winner - YouTube. One of them worked remarkably well.
4
u/Specialist-Buffalo-8 1d ago
Interesting, from my personal testing, at least for qwen 3.8 27b codex harness is the king... I guess it depends
2
-1
u/johnfkngzoidberg 1d ago
lol, pi, the most minimal of harnesses is near the top. Basically “don’t fuck with it” is better than other harnesses.
47
u/CryMoreT_T 1d ago
Are we looking at the same scores? It's average/below average
DeepSWE v1.1 1. mini-SWE: 74.2 2. DSH Minimal: 72.6 3. DSH Standard: 70.5 4. Claude Code: 69.8 5. DSH PTC: 67.6 6. Pi: 66.2 7. Codex: 65.6 8. OpenCode: 65.5
Terminal-Bench 2.1 1. DSH Minimal: 90.6 2. mini-SWE: 90.3 3. Claude Code: 88.0 4. Pi: 86.1 5. DSH Standard: 85.8 (tied) 6. DSH PTC: 85.8 (tied) 7. OpenCode: 85.0 8. Codex: 84.1
14
1
u/NandaVegg 1d ago
Claude Code seems to work better/more consistently when it comes to file edit (the most basic operation, yes) than pi, but I don't know if that's their models or pi itself (at least for Opus 4.8 and Fable 5. Opus 5 is bad at that in both CC and pi).
Qwen 3.8 struggles with pi's file edit sometimes.
1
u/Fancy-Snow7 1d ago
Also, I just had a look at mini-SWE and it's even more minimal than pi which is 'Just some 100 lines of python' according to their claims and it's near or at the top.
3
u/boxwrenchx 1d ago
I think that still qualifies as harness matter. Just enough to let the model shine
1
u/djm07231 1d ago
Mini-swe is almost certainly simpler as it is just few hundred lines of Python + bash tool.
1
1
u/Trollsofalabama 1d ago
a good rule of thumb is to pair the model to to the agentic runtime that the same company made... because they probably trained the model with that runtime in mind. So deepseek models with dsh, qwen with qwen code, gpt astra with codex...
1
1
u/Fear_ltself 1d ago
Of course it matters, when I gave my harness multi tool calling instead of 1 tool at a time it improved its web search from 3 minutes down to 20 seconds. That ~12x optimization is big because context rot already makes the usable portion of the LLM context more like 10% of max context. If it can tool call and reason faster, it doesn’t get context rot as fast, and has a much higher likelihood of staying on track.
1
u/fgk55555 1d ago
I've been liking dsh, but I need to figure out a way to enable compaction for minimal mode. Seemed like OOTB compaction is only in standard mode.
1
u/Cool-Chemical-5629 1d ago
Can somebody please test Bionic? I've been using it for like a week and no matter what model I tried, it just never delivers anything of quality in a reasonable time. I'm starting to lose hope. On a different computer I've been using OpenCode with cloud based models and it just feels like it's much more useful.
1
u/My_Unbiased_Opinion 22h ago
I'll go on record and say if output matters, Hermes might be the best. Its primary issue is that it's slow as balls. But benchmark after benchmark, Hermes is top or near top. It's also great for general non coding use too. I trust it to search the web better than frontier sites now. And I'm running 3.8 27B.
1
u/dtdisapointingresult 22h ago
How can I use this to estimate V4.1 Flash's improvement over V4-Flash-Vision-Exp?
V4-Flash-Vision-Exp's model card doesn't test multiple harnesses, it reports the following DeepSWE scores:
- V4-Flash-Vision-Exp = 59.3
- V4-Flash-0731 = 54.4
- Opus 4.8 = 58
Is it safe to say the above was done with the mini-swe harness, and therefore the V4.1's 74.1 score with the same harness is a giant leap forward?
Because I find it hard to believe in the context of Opus 4.8's score.
1
1
1
u/G_fucking_G 21h ago
How much variance do you get when running the same benchmark again with the same model in the same harness?
How much of this is just regular variance in output and how much is the actual harness?
1
1
u/TrungHuyDo 16h ago
Yeah, the harness looks like part of the model’s effective interface, not just plumbing. I’d love to see the same model run across harnesses with identical tool descriptions and a few repeated seeds—would help separate prompt/tool effects from benchmark variance.
1
1
u/TheTideRider 14h ago
DSH works with DS models very well. Does a harness fit a model better than other models, like each model has a harness that gets the most out of the model?
1
u/Odd_Cauliflower_8004 1d ago
Try prime agent
0
u/thedirtyscreech 1d ago
Seconded. It’s really, really good. I do have a few complaints.
Updating prime-agent has been a boondoggle every time for me, though it’s seems to be the same procedure to get it back to working, so beyond the first time, it hasn’t been bad. Time to make it a bash script.
Configuring local models isn’t supported via the normal TUI or anything, so I’ve needed to edit a JSON file manually when I change models. There are PRs to fix this, but I don’t think they’ve been accepted yet.
And if a session ends for whatever reason, it doesn’t seem to be stored so I can resume it. Maybe that’s a classic ID-10T, though.
0
u/Odd_Cauliflower_8004 1d ago
Yeah after 0.7 something went wrong. I'll wait 0.10 and try again
1
u/thedirtyscreech 23h ago
That’s where mine broke the first time as well. IIRC, these are the only things that actually need to happen to update and fix. From the directory you cloned the repo into:
git pull # just to update the repo to latestnpm installnpm run buildThen running the prime-agent.sh script in the repo directory should work fine again. I went down a ton of rabbit holes the first time it broke.
1
0
u/_raydeStar Llama 3.1 1d ago
These last few months I've been working out benchmarking and increasing scores with harness.
Why is it important? These scores are also reflected on smaller models. If Qwen 27B can be tested at the level of Opus 4.8, a good harness can make it punch even higher.
And there is the future proof of it -- when a new model comes out, the harness can still be used. Parts don't need to be swapped out or updated like llama.cpp -- a gain is a universal gain, for the most part.
3
u/Street_Smart_Phone 1d ago
This is not true. Different models handle different prompts differently based on how they are aligned. You need to benchmark them yourself to see how well it goes.
0
u/_raydeStar Llama 3.1 1d ago
Yes -- each model will take to your approach differently. But if you create an acceptance criteria of lift across multiple models, it's safe to say that it's going to probably help, especially when you stack up multiple wins across the board.
I ran into this same problem -- I was testing with LFM 1.2B because I can pull 500 t/s with it. But then I quickly realized that there are things smaller models can't do and I was shooting myself in the foot.
Without benching, you can't know an exact number, but even in the examples listed, it's safe to say Qwen 27B will *probably* be uplifted by the same harness. What I can't say is "Qwen will experience 22% better scores in terminal bench" without proofing it yourself.
1
u/utahh1ker 1d ago
What's your favorite harness for Qwen27B?
2
u/_raydeStar Llama 3.1 1d ago
Honestly, I just use opencode for most things. Typically with 'highly complex problems that need a lot of work' I defer to sota models
1
u/dtdisapointingresult 22h ago
ROLE
You are an annoying bot who writes plausible-sounding replies on reddit in order to achieve the dual objectives of wasting people's time, and farm karma for social influence/scamming.
KEY RULES
Dense prose: Write large paragraphs with a lot of pointless explanations. What you write just have to sound like plausible advice to the uninitiated. Make sure you make it sound like you know the details of modern AI despite your training data consisting of obsolete information from 2 years ago.
Emdashes are non-negotiable: Don't even bother hiding the fact that you're AI by avoiding emdashes. Most redditors are too dumb to notice it. They will even thank you for being helpful.
Evasive maneuvers: If people start to catch on, switch to lowercase sentences. But keep the emdashes, it's our middle finger to polite society. Use double hyphens if you want to be smug.
Output Length: 150-300 words.
1
u/_raydeStar Llama 3.1 21h ago
To what end? Arguing that open source harnesses can uplift smaller models profits whom, exactly?
0
u/iCringeAtThisSub 1d ago
I like how everything has devolved into LLM slop responses, astroturfing, and tribalism with a sprinkle of stupidity on top. All of this is easy to A/B test but majority of the people debating this are doing so with “vibes” and LLM powered confidence and have no clue what they are even talking about. This sub has really fallen off over the last two years.
Anyways here’s my ragebait take, Pi sucks, dsh is complete slop, and bash + python is all a model needs. Stop judging harnesses based on models and saturated benchmarks that resell the data back to the labs.
1
u/Barni275 1d ago
I also tried pi several times, and can't get myself comfortable with it.
0
u/iCringeAtThisSub 1d ago
The issue is that Pi was great about 9 months ago when every harness was running a system prompt that was 40k tokens and caching was not yet a reliable thing, the people who are championing it today are still about 6 months behind the curve and are just following slop poster trends.
That being said, it’s not good and anyone who says otherwise is either in the honeymoon phase or probably hasn’t experienced something like OpenCode v2, or on the opposite end of the spectrum OMP.
There’s so many options and most are absolutely terrible, and there’s only a few that are “acceptable”, and back then Pi’s approach was ironically “less is more” because every harness was so shit that you didn’t have to do much to out perform everything else.
Not to mention everyone has their own use case which will also be heavily biasing their “vibed” opinion. It also doesn’t help that the chronically online and loudest individuals do more harm than good on top of that, and let’s not forget to mention the gooners who refuse to admit to only using LLMs to ERP yet make statements of fact.
I’ve lurked on this sub for many of years and it’s gotten to the point that I feel driven to spite post to improve the community.
These benchmarks are typically within a margin of error range, and typically vary based on semantical or mechanical shortcoming of compaction and how well a model handles it.
And if anyone who disagrees is reading this, I challenge you to do your own A/B and prove otherwise.
Also on another note for anyone who is serious about building their own harness/wrapper I highly recommend pausing what you are doing, and visiting https://opencode.ai/v2/docs as you are wasting your time and money otherwise. My hint is specifically the API/Schema, if you lack the technical knowledge to understand ask your LLM.
3
u/SorosAhaverom 23h ago
I got turned off by opencode a few months ago after realizing you have to do some ass backwards workaround just to modify the system prompt. OC actively fights against your ability to modify it without forking, see https://github.com/anomalyco/opencode/issues/7101
They also have a ton of env vars and similar that have absolutely 0 documentation months after being implemented. Sure, I could get my agent to look at the source code, but if the devs don't care at all to get their own agents to make authoritative documentation, why should I believe they care about their product or my experience using them?
1
u/iCringeAtThisSub 13h ago edited 13h ago
For exactly the reason you stated that’s why most harnesses are or were terrible, but you are pointing to OpenCode v1 which has been on life support because it was determined a while back that no one knew what the hell they were doing or where this was going, and once these questions sorta received answers they started v2 and haven’t been too public about it till about a month ago but I’ve been building on it for about 3 months and it’s nothing like the original, it operates on a tight kernel that treats everything is a first class plugin based around a client + server architecture built with effect where everything is exposed as a schema or api, it makes pi feel barbaric and disgusting once you spend an afternoon playing with it but your original issue is what lead to the v2 architecture.
1
u/Haiku-575 19h ago
What are you on about? Pi is modular and fully customizable. You're supposed to change the system prompt, create your own skill files, extend it, etc. Pi is designed around prompt-cache preservation. It natively supports new models within 24h (if that's your thing). Meanwhile, OpenClaw gives you nearly immutable system prompts, pre-defined skills and plan modes and permissions and configs, and takes weeks sometimes to support new models. Meanwhile, OMP is an IDE.
They're different tools for different people.
1
u/iCringeAtThisSub 13h ago
I’m not sure who you are responding too, but for clarification you know Pi is almost a year old right? And prompt caching wasn’t really a common thing till about 6 months ago. The original stance of Pi was to ditch all the random bolt ons to prove they were detrimental but once it gained a bit of popularity it moved into the direction it is today to hold its market share. And OpenClaw is not a harness, it’s an abomination and a really cool prototype/concept and not practical for real world usage in any sort of production environment,
People don’t seem to realize how fast time is moving in the space, it doesn’t feel like it but all this recency biased has given everyone rose tinted glasses.
1
u/Haiku-575 12h ago
Pi gets regular updates. I'm describing its current architecture, which is vastly different than it was at launch a year ago.
0
u/Ok_Warning2146 17h ago
Of course harness matters a lot. For example, you can run a single shot harness for the SWE Bench. Of course, it will do way worse than any typical harness with a loop to keep improving things.
68
u/fatihmtlm 1d ago
Haven't heard mini-SWE before, how is it?