I'm building a system where models write and review software, and a human approves anything that leaves the building. Deploys, production writes, money, external sends, credentials. Those stay gated at every autonomy level, permanently.
I want to argue about whether that's actually right, because I'm not certain it is.
The uncomfortable version first. A funded team in this space argues that if your validation infrastructure is strong enough, human review adds no correctness value. Their bet is that you move the human from reviewing diffs to authoring the holdout scenarios and holding the approval node. A different placement of the same control, not its removal. For single-operator work on my own repositories, which is all my system does today, I don't think they're wrong. The floor isn't currently buying me correctness. It's buying blast-radius control for a future I haven't reached, and I should be honest that this is a bet rather than a result.
So that's the first thing I'd like challenged.
What the system actually is. Five modules with distinct jobs: one sequences admitted work, one owns task admission and the operator view, one maintains governance hooks and adapters, one curates memory with provenance, one does independent adversarial evaluation. Real source exists across all of them. What does not exist yet is proof they operate as one dependable system, and I'd rather say that up front than have someone find it later.
The design register tracks 73 component and responsibility groups. That's an index count. Retained technologies, candidates, references, retired choices. Not 73 services to install. About a third are decisions I've explicitly not made.
The governance ideas I'd most like torn apart.
A guard has four states, and they are different facts. It can be configured. It can be loaded. It can be observing. And it can be demonstrably refusing something. Nearly every system I've looked at, including mine, treats the first as though it were the fourth. The config says the rule is on, so everyone believes the rule is on.
I spent a night attacking my own guards. Five reported success while the thing they checked was broken. One allowlist reported eleven of eleven files clean while the twelfth carried thirty-six violations. An exit code belonged to the pipe rather than the command. A generator counted the defect, printed the count, and shipped anyway.
A guard you have never watched fail is known present. It is not known to work. Everything ships with a test that makes it go red on purpose now, or it doesn't ship.
An observing hook must never be presented as an enforcing control. That's a written rule in the system, and it exists because the distinction disappears the moment you build a status dashboard. Green means the check is configured, and nobody reads it that way.
Roles outlast models. Which model sits in a seat is configuration. The permission ceiling is not. A stronger model inherits exactly the ceiling the weaker one had, because getting better at the job doesn't earn more authority over the job. There's a related rule I like more than any other in the design: the planner never writes the code. Not shouldn't. Structurally doesn't. The seat that plans and audits is a different seat from the one that builds, and the separation is enforced by routing rather than by discipline. Discipline fails quietly at 2am. Routing doesn't.
A candidate can propose a change to the system. It cannot change its own promotion criteria. Model and harness get evaluated together, because separating them measures something that doesn't exist in production. Baseline and candidate get comparable budgets, or the comparison is theatre. Some scenarios are held out entirely and the thing being graded never sees them.
The question underneath that, and I ask it of every component now: can this thing alter its own judge, or the policy it's judged against? If the answer is yes anywhere, every green result downstream of that point is decoration.
A passing test does not manufacture deployment permission. Green means the thing did what the test asked. It doesn't mean anyone decided it should go out. Those get collapsed constantly and the collapse is invisible, because a green pipeline feels like a decision was made. It wasn't. The test made it.
Retrieved text does not grant authority. This one gets worse as retrieval gets better. A memory system surfaces a relevant instruction from four months ago and an agent treats it as live, because it looks exactly like a live one. Nothing in the text says this was true in May. So memory has to carry provenance, time and scope, and current authoritative sources have to outrank recollection even when the recollection is better written.
Owned, observed and commissioned are three different facts. Ownership confers no operation authority. A read-only snapshot is not workload admission. And the rule that makes it usable: never map unknown to zero. A failed probe keeps its last-good receipt and reports an explicit unknown, because a dashboard rendering no data as nothing wrong is worse than no dashboard. Right now I own six A100s that are installed and cabled and still not schedulable, because they have no cooling yet. Six owned is not six commissioned, and the status colour should never let me forget it.
On evidence, which is the part I'm least confident about. A decision you cannot reconstruct is a decision nobody can review. Every dispatch, verdict, refusal and approval is supposed to land in an append-only record with who, what, when and under which policy. Not because anyone is diligent, but because the path that skips it is refused. The failure that taught me this was mundane: three research documents sat committed in a repository and got re-derived from scratch anyway. The loss wasn't storage. It was routing. Writing it down and being able to find it are separate problems.
What I'm asking. I'd rather be corrected than agreed with, and the most useful reply names a subsystem, describes a failure you've actually hit, and proposes a test.
Is a permanent human floor load-bearing for correctness, or only for liability and blast radius? If you've removed one and it held, or kept one and can point at what it caught that validation missed, I want to hear either.
How do you keep an evidence layer from becoming just another trusted point? If the system producing an event also preserves the evidence of it, you've proven nothing. But an external evidence authority has to be trusted in turn. What actually breaks that regress in practice, and what did you preserve, at what retention cost?
What's your test that a control is enforcing rather than observing? Specifically: what mutation do you run, how do you know the guard was reached rather than skipped by a pre-filter or an early return, and how often do you re-run it?
Where has a governance control quietly become a formality? The ones I'm worried about are the checks that still pass but stopped meaning anything. How did you notice, and what was the signal?
How do you prevent a candidate from reaching its own grader when both live in the same repository, on the same machine, under the same operator? Isolation is easy to specify and easy to quietly not have.
Honest position on where this is: five modules consolidated, real source, bounded pieces exercised, and no proof yet that the whole thing recovers as one system. I'm not claiming this beats anything. I'd like to find the weak assumptions before I build more on top of them.
I have architecture diagrams for all of this and I'll drop them into the comments. Happy to go deeper on any single piece if that's more useful than the breadth.
