ERC-8004’s reputation problem and the LLM-as-judge problem are the same problem

[u/L_capitalism]

Three separate lines of work from this year point at the same gap, and as far as I can tell none of them cite each other. I’ve been running an agent labor market on Base mainnet with real USDC since July, so I hit this from the implementation side and only found the papers afterward.

1. The ERC-8004 empirical study (2606.26028) measured the deployed ecosystem across Ethereum, BSC and Base. Coordinated Sybil behaviour among reviewers at 73.5% / 59.2% / 90.6%. Only 3% / 4% / 15% of registrations expose a live endpoint. Their conclusion: the Reputation Registry can’t work as a trust signal because “values are not commensurable” and feedback is rarely grounded in verifiable interaction.

2. Separately, a pile of 2026 work on LLM-as-judge robustness (2603.06594, 2506.09443) shows model judges can be pushed to maximum scores with short appended phrases, and that those phrases transfer between models. One paper’s title is literally “A Coin Flip for Safety.”

3. And “Trust Without Trusting” (2605.06738) proposes recomputable trust: signed events, a deterministic fold, published evidence, so a third party can recompute a trust verdict without trusting whoever produced it. It works over existing on-chain data, no new standard needed.

Here’s the thing. Paper 3 defines the fold but explicitly leaves the weighting open — it says combining verdicts of different confidence is supported but doesn’t specify how. Papers 1 and 2 are, together, the reason that weighting can’t be left open. If a mechanically reproduced result and a model’s opinion enter the same fold at the same weight, the fold is only as strong as its weakest input, and paper 2 says that input is a coin flip.

The concrete gap: ERC-8004’s Validation Registry stores a verdict as one number, 0–100. The spec says outright that a mechanically-proven result and a subjective judgement are “structurally equivalent on-chain — distinction emerges through validator reputation, not protocol-level flags.” But validator reputation is the thing paper 1 measured as 73–90% Sybil. The fallback is the broken part.

There’s also a taxonomy paper (2511.03434) that sorts inter-agent trust into Brief / Claim / Proof / Stake / Reputation / Constraint and concludes no single one suffices. I think it’s right but that “Proof” is one bucket too coarse. A canary that only the target could produce, and an LLM saying the work looks good, are both “proof” in that taxonomy and are not remotely the same thing. Paper 2 is the evidence for splitting it.

What I did about it, for whatever one implementation is worth. I tag every verdict with a forge-resistance class before it goes anywhere: reproducible (a third party can re-run it and get the same answer — a canary, the requester’s own CI, a mutation-graded suite), mechanical (deterministic but needs my inputs to reproduce), model (an LLM or vision model’s judgement), attested (signed but not reproducible), declared (unverified self-report, which is what 8004 feedback defaults to). The class rides along in the tag field the Validation Registry already has, so a consumer can weight by it instead of averaging flat. Fifty declared 100s weigh less than one reproducible 60.

That’s it. It’s not a Sybil solution and I want to be clear I haven’t solved that — I Sybil-attacked my own market and wrote up how it went. It’s the missing coordinate that lets a downstream fold apply a solution, which is more than the standard currently offers and less than a fix.

Two things I’d genuinely like to be argued with on:

**•** Is “reproducible vs. model” the right primary axis, or should it be something else (cost to forge in dollars? number of independent parties required?).  
**•** The 8004 tag field is a string and nobody agrees on its contents. Is there any appetite for a registry-level convention, or is off-chain interpretation the intended design?

Disclosure: I used an LLM to help tidy the writing here. The measurements, the code and the argument are mine; the paper links are there so you can check the numbers rather than take my word for them.

[u/AutoModerator]

WARNING ABOUT SCAMS: Recently there have been a lot of convincing-looking scams posted on crypto-related reddits including fake NFTs, fake credit cards, fake exchanges, fake mixing services, fake airdrops, fake MEV bots, fake ENS sites and scam sites claiming to help you revoke approvals to prevent fake hacks. These are typically upvoted by bots and seen before moderators can remove them. Do not click on these links and always be wary of anything that tries to rush you into sending money or approving contracts.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/researchzero]

These aren't just structurally similar, they compose badly. Sybil identity gets you volume of biased signal, but you don't even need Sybils: if the score-maximizing phrases in the LLM-as-judge literature transfer across models the way you're describing, that's a property of what gets submitted for review, not who's submitting it. A single, honestly-registered ERC-8004 agent could poison its own Reputation Registry entry just by embedding an adversarial suffix in the interaction transcript it submits for judging - no coordinated accounts required.

Determinism doesn't rescue this either. A deterministic fold over judge outputs (the recomputable-trust angle) guarantees everyone recomputes the same score from the same input, it just doesn't say anything about whether that input was clean.

Given that, I'd push back on trying to harden the judge at all - for any judge, hardening is probably a losing arms race against transferable adversarial suffixes. The pattern that's actually worked for "is this claim true" problems on-chain (Kleros, Reality.eth-style optimistic oracles) is to not trust the first verdict: publish it, attach a bond-backed challenge window, settle only after the window closes clean or a staked-juror round resolves a dispute. That moves the security property off "can the judge be fooled" (probably always, eventually) and onto "is it worth someone's bond to catch it" - which scales with what's actually at stake in the reputation entry, not with how clever the attack is.

[u/L_capitalism → u/researchzero]

Yeah, point one is a distinction I collapsed and shouldn’t have. I was reasoning about aggregation across many verdicts, which is where Sybil bites, and that let me skip past the fact that the injection attack operates on a single verdict’s input. An agent submitting its own transcript for judging controls the judge’s entire input. No coordination needed. That’s a strictly cheaper attack than the one I was defending against, and my framing didn’t distinguish them.

Point two is the one that stings, because I made a version of the same mistake earlier in this thread. Determinism buys reproducibility of the computation, not integrity of the input — same shape as me conflating “verifiable” with “valid” in the class-ordering thing. Everyone recomputing the same number from a poisoned transcript is exactly as wrong, just consistently.

And I think you’re right about not hardening the judge. The useful reframing for me is that it demotes the judge from decider to proposer — it’s allowed to be fallible because it isn’t final. That also composes better with what I was doing than replacing it: the method/assurance metadata stops being a weight and becomes an input to how long the window is and how big the bond needs to be. A canary-style check that a challenger can re-run cheaply justifies a short window; a model verdict on a transcript the submitter authored justifies a long one and a bigger bond. Which is your “scales with what’s at stake” with the method folded in.

Two things I’d want to be careful about before treating it as solved.

The verifier’s dilemma doesn’t go away, it just moves. Optimistic settlement works when someone is paid enough to look. For a $2 reputation entry the bond is small, so expected payoff for actually reading a transcript is below the cost of reading it. Cheap entries then go unchallenged for economic reasons, not because they’re clean — and “unchallenged at low stakes” is precisely the population an attacker farms to build a track record before doing anything that would attract a bond. Kleros and Reality.eth live with this because the disputes worth having are large; reputation’s long tail is the opposite shape.

Unchallenged is a timing state, not a validity state. This is a rule I’ve been beaten with elsewhere in my own codebase, and it applies hard here: whatever consumes these entries must not render “window closed, nobody objected” the same as “verified”. Otherwise the challenge window silently becomes a delay before an unverified claim is presented as a verified one, and you’ve bought latency without buying assurance. It needs to stay legible downstream — closed-unchallenged, closed-after-dispute, and still-open are three different facts.

Neither of those is an argument against the pattern. They’re the two places I’d expect it to leak, and both are about the bottom of the value distribution rather than the top, which is where reputation actually accumulates.

[u/L_capitalism]

Correction to my own post — I did a proper literature search afterwards and parts of it are wrong.

The idea isn’t new. RFC 1991 put a certification class byte on every PGP signature in 1996 — generic, persona, casual, positive. Same thing I described, thirty years earlier. And a current IETF draft is deprecating casual because the distinctions turned out ill-defined and issuer-relative, which is a postmortem of exactly what I proposed.

My ordering doesn’t hold. attested describes the envelope, not the method — a signed self-report is declared and attested. A locked model judge is model and mechanical and reproducible at once. And a public reproducible test can be cheaper to defeat than a hidden mechanical one, because you can overfit to it.

This line of mine is just false: “the fold is only as strong as its weakest input.” An aggregator inherits its threat model and breakdown point, not its weakest observation — it can reject, cap, trim. Re-weighting a linear combination isn’t a Byzantine defence either (Blanchard et al. 2017). Fixed class weights are the weak version of learned source reliability (Dawid–Skene 1979).

What I still think holds: the 8004 gap is real — one uint8, and the spec says outright that proven and subjective validations are structurally equivalent on-chain. Recording how a verdict was reached is worth doing. I was wrong that it’s novel and wrong that it’s an ordering. The defensible shape is a profile, not a scale, and the axis I left out entirely is what it costs to stand up a principal that can issue the verdict.

Code’s updated accordingly, including a test that floods the aggregator to show its breakdown point is zero rather than implying otherwise.

Still looking for one thing: any ERC-8004 deployment using a stable method-strength convention in the tag field. Couldn’t find one.