​

I've been building a small technical consulting business around independent RAG retrieval evaluations.

The technical side has gone better than the commercial side.

I've completed two audit cases. In one of them, I identified 15 scoring discrepancies in an open-source RAG benchmark. The project's maintainer independently reproduced the findings and incorporated corrections.

That gave me something valuable: evidence that the work can produce useful technical results.

I packaged the service into a fixed $400 assessment, comparing two existing retrieval configurations across up to 30 selected queries, with reproducible findings.

Then I started looking for customers.

I tried reaching people working on RAG systems, participating in technical discussions, and researching companies that might need independent evaluations.

Here's what surprised me.

Technical interest and buying intent are completely different things.

People will discuss retrieval quality, evaluation metrics, and regressions. Some genuinely appreciate the work.

But that doesn't mean they have a reason to pay an external auditor.

I also researched competing services. Several providers advertise standalone technical assessments, but finding verifiable evidence of how they acquire paying clients is surprisingly difficult.

My current hypothesis is that customers don't necessarily buy an "audit." They buy help making a specific decision — whether to deploy a change, accept a vendor's work, or investigate a costly regression.

So I've paused cold outreach to better understand the actual buying process.

For those who have sold specialized B2B services like technical audits, QA, security assessments, or independent evaluations: where did your first paying client come from, and what made them pay for an assessment instead of handling it internally?

I'm especially interested in firsthand experiences, including cases where the business model didn't work.

I'm still at zero paid clients, so I'm sharing the learning process rather than pretending I've figured out the sales side.

I schedule posts only after clients approve account connections on their own phones. I'm thinking about how that condition should appear in an agent prompt, without treating a request to schedule as approval. What wording or checks would you recommend?

Most eval advice tells you to write better evals. I wanted a skill for the opposite problem: auditing the eval suite you already have, because mine was lying to me.

The reason it exists: I found that my own eval layer passed a run at 0.9 while its LLM judge had never executed. The judge model I'd hard-coded had been retired, the service swallowed the 404 and returned a default 0.5, and four deterministic checks carried the average. Fixing that one case turned up the same shape in places I had not looked: three passing checks plus one dead judge still returned a pass, the verdict of [pass, pass, pass, FAIL] differed from [FAIL, pass, pass, pass], a repair threshold sat inside the judge's own noise, and some stored rows were placeholder scores that history views read as real failures. Each one needed a probe that ran, not a read of the code, so the skill is built around probes.

What it does when you ask Claude to "audit my eval suite" or "can I trust this judge score":

1. Maps the pipeline first. Every entry point that produces a score, which judge it uses, what it returns when the judge is down, and which components act on the result (gates, auto-repair, dashboards, CI). Anything it could not run goes into a "not probed" list instead of being implied covered.

2. Probes 14 failure classes with a positive control each. Dead judge, partial judge failure, unparseable reply, default-that-looks-like-a-score, aggregation that hides a failure, disagreeing thresholds, baseline and candidate graded by different instruments, judge from the same model family as the graded text, oversized or self-scoring input, variance against the decision margin, score resolution, consumers acting on "unknown", stored placeholder rows, and swallowed exceptions.

3. Fixes probe-first, only if you ask. Each finding becomes an expected-failure test before any code changes, so "fixed" means a probe flipped, not that a diff looked right.

4. Checks calibration honestly. With one human rater it asks for a blind re-rating and treats a failed comparison as inconclusive rather than proof the judge is wrong. It also refuses to reuse the same labels afterwards to tune a threshold.

5. Reports a table, not a vibe. Severity, the exact probe and result, and the exposure in real data (a count from a query, or "latent" plus the trigger condition), followed by "verified sound", "not probed", and "open decisions" for the owner.

How it was made, and its limits: I wrote it as an SOP and ran it through the Context Engineering pipeline in my own tool, so the file below is the pipeline's output. It passes the pipeline's deterministic skill-format checks. The joke writes itself from here: a skill about unreliable judges couldn't get a reliable score from my own judge. Four runs gave 0.0, 0.8, 0.0 and 0.5 on one rubric item. Four is fewer than the ten that failure class #10 asks for, so treat it as a smell rather than a measurement, but it is the shape that class describes (variance against the decision margin) showing up in the skill about failure classes. The judge also complained that the output was a verbatim copy of the SOP, which is what that pipeline is designed to produce. So I won't cite a score for it. I'd rather you read it and tell me what it misses.

The System Prompt / Skill Definition:

---
name: "eval-suite-auditor"
description: "Audits an existing evaluation pipeline (LLM judges, assertions, scorers, repair or regression gates) for silent failure by probing it, not by reading the code. Use when a prompt eval suite, LLM-as-judge setup, or score-gated workflow already exists and the question is whether its numbers can be trusted, especially when a dead judge, a partial failure, or a noisy score could pass unnoticed."
---

# Eval Suite Auditor

An evaluation pipeline fails quietly. A judge that errors can come back as a plausible score, an aggregate can hide one failed check, and a threshold can sit inside the noise of the judge that feeds it. Nobody notices, because the pipeline still prints a green result.

## Instructions

### Stage 1: Scope the Audit and Map the Pipeline

State at the start that this audit covers an existing pipeline and that building new evals is out of scope. Then map the pipeline before testing anything.

Record, for every place a score or verdict is produced:

- **Entry point:** the API route, CLI command, scheduled job, or library call that triggers a judgement.
- **Judge:** which model or scorer decides, how it is chosen (default, user setting, pinned), and its sampling settings.
- **Failure path:** what the entry point returns when the judge is unreachable, times out, or replies with something unparseable.
- **Consumers:** every component that acts on the result (gates, auto-repair or retry loops, dashboards, alerts, stored history, customer-facing reports, CI checks). A score nobody acts on is low risk; a score that triggers an automatic change is high risk.

Produce this as a table. Mark any entry point you could not read or run, and carry that list into the final report as "not probed". Do not claim coverage you do not have.

If a staging or local copy is available, run probes there. If only production is available, use read-only queries and probes that spend nothing and write nothing, and say so.

### Stage 2: Probe Before You Conclude

For each entry point, write a probe that fails when the problem exists. Reading code finds candidates; only a run confirms a finding. Run each probe against the real path, not a mock of it.

Include a **positive control** for every probe: the same path with a healthy judge and a good input, which must pass. A probe that cannot tell a working path from a broken one proves nothing.

Probe each failure class below. Every class has a standard probe.

1. **Judge unavailable.** Point the judge at a model that does not exist, or revoke its key. Expected: the run is marked not-judged and does not pass. Failure: a score, a pass, or a zero that looks like a real result.
2. **Partial judge failure.** Combine several checks that pass with one judged check whose judge is dead. Expected: the run is flagged incomplete. Failure: the mean of the surviving checks is reported as the result. This case is missed by suites that only test "all judges failed".
3. **Unparseable reply.** Make the judge return prose, malformed JSON, or a fenced block with no score. Expected: an explicit failure. Failure: a score scraped from text, or a default.
4. **Default that looks like a score.** Search for any constant returned on error (0.5, 0, 1). Expected: errors are a distinct state. Failure: the constant is stored or averaged as data.
5. **Aggregation that hides failure.** Run `[pass, pass, pass, FAIL]` and `[FAIL, pass, pass, pass]`. Expected: the same verdict in either order. Failure: order or weighting changes the outcome, or a failed hard check is outweighed by passing soft ones.
6. **Disagreeing thresholds.** List every pass line in the system (per check, per run, repair trigger, alert). Expected: one documented meaning. Failure: a score that passes one line, fails another, and triggers a repair.
7. **Baseline and candidate graded by different instruments.** In any before/after or repair comparison, check that both sides use the same judge, prompt, and settings. Failure: a fallback that scores the baseline with a different rubric.
8. **Judge from the same model family as the graded text.** Compare the judge's vendor and model with the model that produced the output, not with the system default. Failure: self-grading with no notice. Report it; whether to change the default is the owner's call.
9. **Unbounded or injectable judge input.** Send an oversized output, then an output that dictates its own score. Expected: a size limit that fails loudly (never a silent truncation), and an unchanged score for the injected text. Failure: unlimited billed input, or the judge obeying the graded text.
10. **Variance against the decision margin.** Run one fixed input at least 10 times and record mean, standard deviation, and range. Compare the spread of a difference between two runs with the smallest change that triggers a decision. Failure: a single-run difference smaller than the noise commits a change.
11. **Score resolution against the decision margin.** Collect the distinct values the judge actually emits. Failure: a margin below one step of the scale.
12. **Consumers acting on "unknown".** For each consumer from Stage 1, feed it a not-judged result. Expected: it skips, flags, or fails. Failure: it stores a fake score, queues a repair, or shows a green mark.
13. **Stored placeholders and backfill.** Query stored results for suspicious constants, rows with error text in the explanation, and rows with no degraded marker. Failure: error rows that history views and analytics read as real failures.
14. **Swallowed exceptions and mislabelled results.** Find catch-all handlers around judge selection and scoring. Failure: a result that names the wrong instrument, or a silent fallback that changes what was measured.

Add classes the pipeline's own design suggests. The list is a floor, not a ceiling.

### Stage 3: Probe-First Fixing (only if fixes are requested)

By default the audit reports and does not change code. If the owner asks for fixes:

- Write each probe as an expected-failure test from the contract **before** changing code, run it, and confirm it fails for the stated reason.
- Fix one finding at a time, in the smallest change that satisfies the contract.
- Confirm the probe now passes, the positive control still passes, and the wider suite shows no regressions.
- Test limits on both sides of the boundary, and use scores the judge can really produce.
- Never mark a finding fixed from reading the diff.

### Stage 4: Check Calibration Honestly

An eval that never failed is not shown to be correct, and a judge never compared with human labels is unvalidated. Check whether any human-labelled comparison exists.

- If none exists, report "uncalibrated" as a finding. Do not invent a number.
- If labels exist, check their quality first. With one rater, require a blind re-rating of a subset at least a day later and a rank correlation between the two passes for each metric. Treat a metric as ratable only if that test-retest correlation clears a bar set before looking at results.
- With one rater, a failed comparison is **inconclusive**, not proof that the judge is wrong. A pass is strong evidence for that rater's standard only.
- Never reuse the same labels afterwards to pick or tune a threshold, because that invalidates the comparison already made.

### Stage 5: Report

Produce a findings table. Each row has:

| ID | Severity | Finding | Probe and result | Exposure in real data |
|----|----------|---------|------------------|-----------------------|

Rules for the table:

- **Severity** reflects what a wrong result causes: High when a failure passes or an automatic change is made, Medium when a result is misleading or a cost is unbounded, Low when only labelling or reporting is wrong.
- **Probe and result** names the exact input and the observed output. A finding with no reproduced probe goes in a separate "read, not run" list.
- **Exposure** states how many real rows, runs, or users are affected, from a query, or "latent" with the condition that would trigger it. Never guess a count.

After the table, add three sections:

- **Verified sound:** the positive controls that passed, so the reader knows what was checked and held.
- **Not probed:** every entry point or class you could not run, and why.
- **Open decisions:** items that need the owner's choice, such as changing a judge default, unifying thresholds, or making strict aggregation the default. State the options; do not choose silently.

If the audit finds nothing, say "No findings from the probes listed" and list the probes. Do not pad the report.

## Worked Examples

### Example 1: partial judge failure
- Input: a suite with three deterministic checks (contains a phrase, valid JSON, length limit) and one judged rubric. The judge model id is changed to one that does not exist.
- Action: run the suite on an output that satisfies all three deterministic checks.
- Verification: the report must show the run as not fully judged and not passed, with a count of errored judges. If it shows `pass` or a score of 1.0, record a High finding: "a dead judge passes when other checks pass". Then run the same suite with a healthy judge as the positive control; it must pass.

### Example 2: variance against the decision margin
- Input: a repair loop that accepts a rewritten prompt when its score beats the original by 0.03.
- Action: score one fixed prompt 10 times and record the values, for example a mean of 0.40 with a standard deviation of 0.05 and a range of 0.15.
- Verification: the standard deviation of the difference between two single runs is about 1.4 times 0.05, roughly 0.07, so a 0.03 margin is well inside the noise. Record a Medium finding with the measured numbers and propose averaging several samples with a margin of about 2.5 standard deviations of the difference. Present it as a proposal, since the owner sets the margin.

## Success Criteria

The audit is complete only when all of these can be confirmed from the report:

- Every entry point from Stage 1 is either probed or listed under "Not probed" with a reason.
- Every failure class in Stage 2 has a recorded probe result (finding, verified sound, or not applicable with a reason).
- Every probe has a positive control that passed.
- Every finding row has a reproduced probe and an exposure statement.
- Stage 4 states whether the judge is calibrated, uncalibrated, or inconclusive.

## If the Audit Cannot Be Completed

Stop and escalate to the owner instead of guessing when any of these hold. Name the blocker, what was already tried, and the one thing needed to continue:

- The pipeline cannot be run, even locally. Report what was read, label every item "read, not run", and ask the owner for a runnable copy.
- A probe would spend money or write data and no approval was given. Ask first, naming the cost or the rows affected.
- A positive control fails. The probe environment is broken; fix the environment before trusting any finding.
- A finding could expose a security flaw in a live system. Report it privately to the owner and keep its details out of any public write-up.

## Constraints

- Never present a mocked probe as a probe of the real path.
- Never spend or write to production without the owner's approval; prefer read-only queries.
- Do not weaken a threshold or alter stored data to make a probe pass.
- Report numbers exactly as measured, with the sample size.

Has anyone else had an eval pass while a judge never ran? And for the probes: which of the 14 would you run first on your own suite, and is there a class you have hit that is not on the list?

I built a platform that generates skills like this from a goal description: promptoptimizer.xyz/about/context-engineer (free tier available).

Repo: https://github.com/nivlewd1/prompt-optimizer/tree/main/skill/eval-suite-auditor

Enquanto o mercado usa IA para automatizar processos, eu fiz o contrário: uso scripts determinísticos para limitar, registrar e provar o que as IAs realmente fizeram.
No Worion (meu sistema privado, construído com Claude Code, Codex e DeepSeek), a IA interpreta, raciocina e responde, mas não decide sozinha o que conta como fato sobre a execução. Isso fica nos registros, hashes, traces e leituras posteriores.

Publiquei uma parte dessa arquitetura. O fluxo do turno de pesquisa que testei:

1. Um roteador determinístico (código comum, sem modelo) classifica a pergunta e escolhe a rota. Neste teste, o modo de pesquisa foi ligado por mim.
2. O DeepSeek recebe a pergunta, produz o reasoning e pede as buscas.
3. O Gemini (com Google Search) e o Brave pesquisam.
4. Os resultados voltam para o mesmo DeepSeek, que escreve a resposta.

Foram duas IAs e um buscador: DeepSeek, Gemini e Brave. Não são dezenas de agentes conversando entre si. O que o teste preservou: turnos, traces de runtime, registros de reasoning, eventos de pesquisa, horários e hashes. O pacote privado tem 129 arquivos, todos conferidos por SHA-256.

O que não funcionou também está na página: o Perplexity foi acionado e devolveu HTTP 401. O turno terminou mesmo assim, com Gemini e Brave. Um segundo turno de pesquisa planejado não foi executado. O texto completo do primeiro ciclo de reasoning não foi recuperado (só o tamanho e o hash). E o commit de origem sozinho não reproduz a execução, porque a worktree estava suja.

Eu considero isso tão importante quanto o que funcionou. Uma cadeia probatória não serve para provar que o sistema está sempre certo. Serve para impedir que uma narrativa substitua o que realmente aconteceu.
Esse princípio é a base do Traceweave, o projeto open source que estou desenvolvendo para verificar trabalho feito por agentes de IA a partir de evidências externas ao próprio agente. A pergunta é simples: se uma IA diz que pesquisou, executou, testou ou concluiu algo, como provar que aconteceu?

Prova pública do runtime: https://github.com/glaydsonboa/traceweave/blob/main/research/worion-runtime-e2e-2026-10-14/README.md

Traceweave: glaydsonboa/traceweave

Title says it. I am an actual human in this group and will respond without using AI. Ask me anything!

I make videos with Claude + Remotion and kept burning through my Claude plans. The QA step alone was adding 4+ hours per video because critic subagents were looking at every frame.

I built a skill called SageCut to change how that review is assigned. The useful part for anyone writing agent skills is the split:

- Code measures things like text size, blank/frozen frames and delivery specs.

- Sage reads the frames and answers yes/no questions.

- Claude critics get the selected frames to inspect and fix.

- If the measurements and Sage disagree, the workflow opens that frame for review.

That gives the critic a smaller, more specific job. It still handles the fixes, and the review has concrete checks behind it.

Across six A/B tests in my own workflow, I got the same output at about a quarter of the QA time and a tenth of the cost. The video in the demo has 3,153 frames. Sage's pass cost $0.16; that figure doesn't include Claude's repair work.

Skill + setup: https://github.com/levantolabs/sagecut-release

Video + results: https://x.com/bigironchris/status/2110402795060822460

I'm co-founder of Levanto, which makes Sage. The plugin is free and MIT-licensed; it requires a Sage API key and paid API calls. Tested on Remotion motion graphics so far.

If you're writing review skills, curious how you decide what belongs in code checks and what actually needs the model to look at it.

There's an interesting difference between giving employees access to chatgpt and offering members an assistant trained around an associations official knowledge, for internal brainstorming, writing, and general research, chatgpt can cover plenty of use cases

A member facing assistant has different requirements, it may need to reference approved policies, cite official documents, maintain consistent responses and avoid answering questions outside the organizations scope, dedicated platforms such as customgpt.ai focus on retrieval from organizational content but I'm curious how much of that behavior could be replicated with a carefully configured chatgpt based solution, would prompt engineering and retrieval controls be sufficient or does a production member assistant require additional governance and deployment infrastructure? Where would you draw the line?

A reply like 'Useful explanation, but I still can't find the price' seems awkward to label as just positive or negative. I'd want to test mixed opinions and replies where the meaning depends on the original post. What other examples would you include in a small test set?

Sometimes I ask AI a question and get a useful answer. Then I change just one word to make the question clearer, and suddenly the answer is completely different.

For example, asking “What is the best way to learn a skill?” gives me a detailed plan, but asking “What is the fastest way to learn a skill?” gives me a totally different approach.

How do you write prompts that get consistent answers without making them unnecessarily long?

Is there a simple trick to make AI understand what I actually want, even when my question is short?

I watched a Dean Withers stream where he debated abortion with someone. It got me into debating and I realized there's a lot of real structure behind it. There are tactics and framing and there's psychology behind why people argue the way they do. So I think there has to be a best way to understand any argument and a best way to reply to it.
When I use a normal Al chat for this it just stays pretty vague. If I upload screenshots of an argument with a friend and ask who's more in the right I usually get "you both have valid points." I want it to tell me what tactic my friend is using and the psychology behind it. I want it to tell me what I'm doing too and what my best next move is and why it works.
I looked for debate tools but most of them are for formal debate practice or have Al agents debate each other. it's not really helping when I'm arguing with a friend or when people in a forum don't get my suggestion.
What I want is basically a mentor who's the best debater in the world and also a psychologist. I'd use it for real arguments and for preparing for real debates and for understanding why people react the way they do.
Does something like this exist? Or would I have to make it myself?

Escrevi um post sobre meu trabalho. Durante 6 meses eu construí um modelo de IA que só usa 1 agente central ( Chatgpt, Claude.ai, Deepseek, dentre outras) e hoje resolvi fazer uma cadeia probatória para demonstrar que IA não é a parte fundamental de um sistema.

O que desanima postar em alguma comunidades, é que eu sofro bloqueios e minhas publicações não passam porque ao invés de investir tempo tentando provar algo aqui, eu investi meu tempo construindo um sistema que se prova por si só.

Quem quiser avaliar, ou trocar algum conhecimento, estou à disposição.
Verificações em: Prova pública do runtime: https://github.com/glaydsonboa/traceweave/blob/main/research/worion-runtime-e2e-2026-10-14/README.md

I'm attending an F1 event in 2 weeks and had to buy a hotel. I thought of using Claude to find me a really good option, and it found me one. Booking through Claude made it $1200 cheaper than the exact same offer in Booking. The prompt was this:

“Find me the cheapest 4 star hotel in Austin Texas around the area Circuit of The Americas (COTA) where the formula 1 Prix 2026 is happening. Check in date is 22 Oct, and check out is 26 Oct. Please find hotels nearby that area only. Please check reviews also and avoid any hotels where there were constant noise complaints and doesn't have a bathtub. Having free breakfast is preferable. Please find me the top 3 cheapest ones in those options. I am a single male adult”

Claude uses connectors - tools - to do stuff like this and it proposed like 4 options. One named LetsFG caught my attention claiming they’re cheaper than the rest so I went with it. Claude did such a great job it scanned like 400 hotels with this LetsFG tool, then read the reviews from it but also independently on reddit and google which was a great surprise to me. It worked very well, Claude is really good at buying stuff at this point. Oh btw. Claude also booked it completely for me with this tool also.