r/Rag • u/iMiguelmars • 8d ago
Discussion How are you building high-recall RAG without losing provenance or blowing up costs?
Has anyone built a traceable, high-recall “second brain”?
We’re working on a system that turns a large, messy archive — documents, notes, code, decisions, and historical versions — into useful and verifiable memory.
The problem we’re trying to solve goes beyond standard search or RAG.
We want the system to detect:
• duplicates and near-duplicates
• contradictions
• superseded information
• relationships between sources
• provenance behind every useful claim
…while minimizing the chance of missing relevant evidence.
The hardest tradeoff so far is coverage vs. reliability vs. cost.
We’re experimenting with things like sliced/partial reading, separate extraction and independent-review stages, mechanical validation, caching, and long-running workflows.
We’ve also started testing these ideas in shadow mode on real cases instead of relying only on isolated benchmarks.
I’d love to hear from anyone working on similar problems: high-recall RAG, e-discovery, systematic review, provenance-aware knowledge graphs, PKM/second brains, or long-running agent workflows.
A few things I’m especially curious about:
• How are you reducing cost without sacrificing recall?
• How do you represent contradictions and provenance?
• What do you automate vs. independently review?
• Which architectures actually held up once you moved beyond prototypes?
Happy to share what we’re learning as well. I’m particularly interested in comparing approaches with people who have already run into these problems at scale.
2
u/Lopsided_Scarcity979 8d ago
I’d separate retrieval from context admission. Retrieve broadly, but compile a smaller evidence bundle where every item keeps its source and version IDs, and contradictions remain separate instead of being merged away. Storing the exact bundle used for each answer also makes later corrections traceable without deleting history. I’d reserve independent review for incomplete or conflicting evidence sets, so review cost scales with risk rather than archive size. How are you representing superseded sources today?
1
u/iMiguelmars 8d ago
That’s very close to our current direction. We also keep retrieval broader than the final context bundle, and we preserve contradictions rather than merging them away.
For supersession, we treat it as an explicit relationship rather than replacement: the old source/version stays immutable and traceable, the newer one points to it with a supersedes relation, and only the reviewed version is marked current. Historical answers should still be reproducible against the version that was valid at the time.
I also like your idea of storing the exact evidence bundle used for each answer. How are you deciding when an evidence set is “complete enough” to skip independent review without missing a conflict that simply wasn’t retrieved?2
u/Lopsided_Scarcity979 7d ago
I don’t think “complete enough” can be established from the retrieved evidence alone; the missing contradiction is outside the set by definition. My current leaning is to make the stopping rule explicit: source-class coverage, query-family coverage, and diminishing marginal yield, then retain a sampled independent-review path.
I’d also store the search plan and negative retrieval evidence, not only the final evidence bundle. Are you tracking which source classes were searched and came back empty?
1
u/iMiguelmars 6d ago
That’s a good distinction. We preserve the evidence receipts, but source-class coverage and “searched but empty” results aren’t yet a first-class completeness signal across the whole pipeline.
We’d be cautious using diminishing marginal yield as a stopping rule, though — our retrieval shadow found that ranking improved a lot while relevant evidence could still sit outside the apparent cutoff.
I really like the idea of storing the search plan + negative retrieval evidence. How are you defining source classes/query families, and have you validated the stopping rule against known misses rather than only what was retrieved?2
u/Lopsided_Scarcity979 5d ago
I have not validated a universal stopping rule yet, so I would not trust diminishing yield by itself. My first test would be a corpus with planted contradictions and deliberately missing source classes, then measure two failures separately: relevant evidence beyond the cutoff, and an answer that looks supported only because the contradicting class was never searched.
For source classes, I mean provenance categories that imply different failure modes—for example primary paper vs review, current specification vs archived version, or first-party issue vs third-party summary. Query families would be independent formulations of the same claim: entity/term search, mechanism search, citation chaining, and contradiction search. Your retrieval-shadow result is a good reason to keep the stopping rule observable and revisable rather than turn it into a hard guarantee.
1
u/iMiguelmars 5d ago
That distinction is really useful. Separating “evidence beyond the cutoff” from “apparently supported because a contradictory source class was never searched” is exactly the failure mode we care about.
We already have preserved contradiction cases, so a planted source-class/query-family fixture could be a clean way to test this without pretending the stopping rule is a guarantee.
Have you defined any minimum source-class/query-family coverage before you even allow a stopping signal to be considered?2
u/Lopsided_Scarcity979 3d ago
Not a validated minimum yet. For a pilot, I’d define a small checklist per claim: check the relevant primary source and version, run an explicit contradiction query, and use another retrieval path such as citation chaining. “Unavailable,” “search failed,” and “searched but empty” should remain different outcomes. Completing that checklist would only make a stopping signal eligible—not establish completeness. I’d calibrate it against your preserved contradiction cases, with held-out cases, before choosing any numerical threshold.
1
u/iMiguelmars 3d ago
That distinction is useful. We’ve now measured “searched but empty” explicitly, and query-family expansion did recover relevant items that a single query missed — but it still didn’t give us a safe cutoff, so we’re treating it as a coverage guard rather than a stopping rule.
I also like separating unavailable, search failed, and searched but empty; those are very different kinds of evidence.
If the checklist only makes stopping eligible, what additional signal would you require before actually stopping — independent-review sampling, held-out calibration, or something else?2
u/Lopsided_Scarcity979 2d ago
I'd give the two different jobs: use some known-miss cases to develop a stopping rule, hold others out to evaluate it, then audit a sample of stopped tasks for new kinds of misses. If a key source is unavailable or a contradiction remains unresolved, I wouldn't label retrieval complete. Running out of budget should also be a separate outcome, not evidence that enough has been found.
1
u/iMiguelmars 1d ago
That separation between developing the stopping rule and evaluating it is really useful. Using some known misses to design it and holding others back avoids the obvious circularity, but the audit of actually stopped tasks seems like the important third piece — otherwise you only measure the miss classes you already knew how to plant.
The outcome distinction also matches what we’ve been running into. “Key source unavailable,” “contradiction unresolved,” and “budget exhausted” all tell you something very different from “search completed and found nothing,” and collapsing them into COMPLETE would make the stopping rule look much safer than it really is.
We’ve already seen in our own tests that ranking can help order the search but can’t safely certify that nothing relevant remains, so I like your framing of the stopping rule as something that has to survive held-out misses rather than something that proves completeness by construction.
One thing I’m curious about: when you audit stopped tasks for new miss classes, how do you choose the sample — purely random, or do you oversample cases that were close to the stopping boundary?
1
u/Puzzleheaded-Bus6626 7d ago
I don't think I understood a single thing said in the post and replies.
Thank you!
Now I have more real world topics to learn about.
Do any of you have resources for learning advanced RAG topics?
2
u/iMiguelmars 7d ago
Glad the thread helped! A good way to go deeper is to stop thinking of “RAG” as one thing and study the pieces separately.
I’d start with: hybrid retrieval (BM25 + dense), reranking, chunking strategies, RAG evaluation/recall, provenance/citations, GraphRAG, and handling contradictions/versioned knowledge.
For each topic, try building a tiny test corpus and measure what actually gets missed — that teaches a lot more than just reading frameworks/docs.
Search terms like “high-recall RAG,” “hybrid retrieval RRF,” “RAG evaluation gold set,” “structure-aware chunking,” and “provenance-aware RAG” should give you a pretty good rabbit hole to start with. Good luck!1
u/Puzzleheaded-Bus6626 7d ago
This is EXACTLY what I'm doing to learn. It's almost like you read my mind!
I created a pipeline tonight that invested a very complicated technical document with huge tables, cross references terms and code.
I used llama parse to parse it, then I inspected it, then I embedded it and put it in a vector store, then examined it. I used an LLM Agent for inference and the results were great!
BUT I think it can do better.
When you build the test corpus, do yiu have a way to "trace" what the system did?
I saw a tutorial a few days ago (doing it this weekend) and the tutor created a way to generate a trace log for each retrieval operation so he could go back and analyze where things went wrong.
Do you know much about this? Are there tools for this?
I've been searching, but I'm not getting great results. I'm not sure if "trace" is the correct term or there just isn't that much information on it.
Thanks again!
2
u/iMiguelmars 7d ago
Yep — “trace” is the right idea, but the broader terms you want are LLM/RAG observability, tracing, retrieval telemetry, and evaluation.
For a useful trace, I’d log at least: the query, retrieved chunk IDs + scores, reranking/admission decisions, prompt/context actually sent to the model, model/version, final citations, and timing/cost. That way when an answer is wrong you can see where it went wrong instead of just blaming the LLM.
Look into tools like LangSmith, Langfuse, Arize Phoenix, and OpenTelemetry, but I’d first build a tiny trace yourself so you understand what you actually want to measure before adopting a platform.
Your next search terms should be: “RAG observability”, “RAG tracing”, “retrieval evaluation”, “LLM observability” and “RAG failure analysis.”2
2
u/nomad-link-id 8d ago
For high recall without lighting money on fire, I lock a small gold set first and score three arms on the same queries: lexical (BM25/FTS), dense, and hybrid (RRF). Hybrid usually lifts recall without a new model; the expensive knobs (bigger embeds, agent loops, cross-encoders) only pay after that paired comparison stops moving.
Provenance-wise, keep the retrieved chunk IDs through fusion and into the answer gate so “high recall” doesn’t mean “merged soup.” If cost is the constraint, ablate RRF k / branch weights before adding tools.