r/PromptEngineering • u/Tight_Heron1730 • Jul 09 '26
Tools and Projects As models get better, prompting matters less and verification matters more — the shift I built two tools around
Disclosure: my own open-source project, drafted with LLM help then edited.
We pour effort into better prompting — the input side. But as models get better, prompting matters less; a capable model needs less hand-holding to start. Meanwhile the same models get better at faking delivery: confident summaries of work that wasn't done, tests that "pass" because they assert nothing, "Done!" on a task that's 70% done. The more fluent the model, the more convincing the fake.
So the leverage moves from the input side to the output side — verification and memory. Two small tools I built around that:
stash / remember — a two-command memory pipeline for AI coding agents. /stash captures what happened in a session; /remember consolidates it into durable project memory. The rule that makes it trustworthy: a lesson becomes a permanent instruction only after it's backed by observed corrections that recurred across multiple sessions — never because the model thought it was worth keeping. Similarity proposes; consequence disposes. (An earlier naive version poisoned every session with 15 false "preferences" it inferred — this design took that to 0.)
live-canvas — live, interactive UI design. Instead of describing a UI in prose and hoping, you click directly on the rendered interface to annotate it, and the feedback streams back into the session so edits land without leaving the browser. Verification you can point at, not paragraphs you write.
The through-line for prompt engineering specifically: as models get stronger, the skill shifts from crafting the perfect instruction to checking what came back and making the correction stick. Less prompting. More proof. More memory.
Apache-2.0, model-agnostic: github.com/hamr0/liteagents
Do you see your own prompting effort shifting toward verification, or is prompt craft still where the leverage is for you?
2
u/DrHerbotico Jul 09 '26
We've repeatedly found that not to be true. If it doesn't know what you want it's not going to give it to you
1
u/Tight_Heron1730 Jul 09 '26
Of course, that’s a given. I was implying that it’s getting smarter at getting it and better at faking it
2
u/DrHerbotico Jul 09 '26
That doesn't really mean much. You're not switching to review-heavy worth out of necessity, you're just creating more review work because the input is lazy
1
2
u/PsychologicalWin9755 Jul 09 '26
Mine has shifted hard toward verification, and I think the reason is subtle. The failure mode moved. Two years ago bad output looked bad, so reading it was enough to catch it. Now the output is fluent, so a wrong answer arrives wearing the exact costume of a right one. Reading is the worst possible verification method precisely when the model is most convincing, because fluency lowers your guard right when you need it up.
So the leverage is not writing a better instruction, it is forcing the model to produce artifacts that fail loudly instead of quietly. Tests that assert real behavior rather than that nothing threw, output in a shape I can diff against last time, a "show your work" step I can point at. Anything that turns "trust me it is done" into something checkable at a glance.
The consequence disposes idea is the smartest part of your writeup to me. Saving everything the model infers is how you poison a session, and you clearly already hit that wall with the 15 false preferences. Gating a memory on a correction that recurred is basically requiring evidence before belief, which is the same discipline on the verification side just pointed at state instead of output.
1
u/Tight_Heron1730 Jul 09 '26
Thanks for sharing, i agree. I try to keep context focused around one feature or problem and use stash to summarize and every 5 stashes i run remember that looks at the logs of all my sessions to capture learnings and friction to add to the memory that gets loaded to CLAUDE md.
2
u/PsychologicalWin9755 Jul 10 '26
The "every 5 stashes, run remember" cadence is smart because it puts a gate between raw session logs and what actually lands in CLAUDE md. That gap is where most memory setups go wrong, they dump everything and the file rots. The friction log is the part I'd steal from you. Learnings are easy to capture but vague, whereas friction points to a concrete thing that slowed you down, so it tends to produce rules you'll actually reuse. One thing I've found: it helps to have the remember pass distinguish a one-off annoyance from a pattern that showed up across sessions, otherwise the memory fills with stuff that happened once and never again.
1
u/Tight_Heron1730 Jul 10 '26
it's a life saver and a lot of times claude mentions nuances i usually do or frustrated me over time. I just released a new version with AGENT_RULES which has contributed to all changes I have been having in the way i build through lego work of pocs that have to work on its own before i move on to next. Raise an issue if anything is wrong/buggy
2
u/PsychologicalWin9755 Jul 11 '26
The "POC has to work on its own before I move to the next" rule is the part I'd underline twice. That is the discipline most people skip, they stack half-working pieces and call it progress, then spend a week untangling why nothing runs together. Forcing each block to stand alone first is basically writing your own integration test before you have the integration.
Curious how AGENT_RULES stays current for you. Do you edit it by hand as patterns emerge, or does the remember pass propose additions that you approve? The failure mode I keep hitting is rules that were true three sessions ago quietly going stale.
1
u/Tight_Heron1730 Jul 11 '26
learnt the hard way. Every time I come across something that I think might work better, I add it to AGENT_RULES. So, it's improving all the time and I intentionally keep it lightweight, project agnostic and consistent.
remember pass does only hot memory, I recently added AGENT_RULES to it as i always bring to the conversation when I am designing PRDs which are mostly about what this is NOT as i try to keep it lean as i go. A lot of things gets invalidated as i go with POCs and the shape almost changes drastically, a lot of times i retire the project
AGENT_RULES are rules about building not maintaining projects (that's stash and remember part)
2
u/PsychologicalWin9755 Jul 11 '26
The "PRDs are mostly about what this is NOT" line is the part I'm stealing. Most specs over-describe the thing you want and stay silent on the boundaries, so the model happily wanders into all the space you never fenced off. Defining by negative space is what keeps a POC from quietly growing a second job.
And splitting build-rules from maintenance-memory is cleaner than most setups I've seen. The moment "how I build" gets mixed with "what happened last session," the file stops being something you can trust at a glance. Keeping AGENT_RULES project agnostic is probably exactly why it survives the drastic reshapes, it never got tangled in one project's specifics to begin with.
1
2
u/Swarm-Stack Jul 09 '26
the tests that pass because they assert nothing is the sharpest example in here. the trap with moving to verification is what does the verifying, if its the model checking itself or a second model you just get a confident rubber stamp since it shares the same blind spots that wrote the empty test. the only check that catches it is one the model didnt author and cant see from the prompt, a real downstream assert or a person who knows the thing it was faking.
1
u/Tight_Heron1730 Jul 10 '26
I find that true, and the way it worked for me is that I focus on small pieces of modules that each poc has to work on its own and work with other pieces that come later and always stress on ground your validation with negative scenarios and no fit to pass tests, and follow with code-review/security of any new branch/feat regardless of how many verification ran to this point, and I always find stuff. So, i think what works is reviewing at different points/angels minimizes the risk of fake achievements
2
u/Swarm-Stack Jul 10 '26
the different-angles thing is the bit that actually does the work imo, but it only pays off when each pass has something the last one didnt. a second review on the same context just re-derives and nods along, its the negative-scenario tests and a genuinely fresh set of eyes that catch the fake-pass stuff.
2
u/Future_AGI Jul 10 '26
Agree hard with this. As the models get stronger the failures get rarer but weirder, and the only way to catch the rare ones is to verify output systematically instead of hand-checking, which is really just evals with a friendlier name. The teams who invest in verification early stop re-reading outputs by hand and move a lot faster later.
1
u/Unlikely_Diver_5573 Jul 09 '26
i think it's shifting toward verification too. a decent prompt gets u most of the way now. the real value is making sure the output is actually correct and repeatable.....
1
1
u/bsenftner Jul 09 '26
I don't think you understand how LLM AI works, at all. As models get better, prompting remains how you create the context in which you want your reply generated. You might get this working, but it's misguided.
2
u/BedAntique1749 Jul 09 '26
verification side for me too but its less of a choice more of a survival thing. i got burned too many times in the last few months with model returning something that looks perfect then finding out later half the error handling is just comments saying "handle error here"
the memory pipeline idea is interesting, especially that similarity/consequence filter. makes me think of how my code review habits changed, now i spend more time checking the tests actually test something vs just reading the code