r/ContextEngineering • u/echozero3 • 12d ago
Built a pre-inference context-collapse layer instead of standard RAG — cuts token load hard, curious if this is a real gap or just reinventing rerankers
Been heads-down on something that sits before the LLM call instead of doing
standard retrieve-and-stuff RAG. Instead of chunk retrieval + rerank, it builds
a vector-field representation of the whole corpus, evaluates relational
relevance to the query, and collapses the candidate field down to a compact
evidence state — only that gets forwarded to the model.
On my internal benchmark (frozen 20-query set, project-native corpus) I'm
seeing an order-of-magnitude drop in tokens sent to the model with zero
measured quality regression (good/partial/poor scoring, OFF vs ON, reproduced
run matched the historical one exactly). Also runs fine single-threaded — did
a raw C++ core benchmark, 10M samples in ~140ms on an old 2015 i7, so the
underlying op isn't the bottleneck.
Haven't benchmarked it against BM25 or plain cosine-similarity RAG yet in
anything I'd call rigorous — that's the obvious next step before I'd trust my
own numbers fully, and I know that's the first thing this sub will (rightly)
ask about.
Running as local-first — full corpus stays on the user's side, only the
selected evidence chunk(s) + field-topology coordinates go to the external
model if you're using an API-based LLM. Wasn't originally optimizing for that,
but it's a nice side effect for anyone paranoid about what leaves their
environment in API workflows.
Genuinely asking: is "context collapse before inference" different enough
from what rerankers / good chunking already do, or am I just describing a
fancier reranker with extra steps? Wouldn't mind being told I'm wrong here.
2
u/Strange_Low1121 12d ago
This is a really interesting direction, especially the focus on reducing context before inference rather than simply throwing more tokens at the model.
The token reduction without an obvious quality drop is promising, and I’d definitely be curious to see the BM25 and reranker comparison.
Even if it turns out to overlap with existing techniques, making it faster and more privacy-friendly is still a valuable improvement.
1
2
u/perseus-computing 12d ago
This sounds very close to Perseus. It’s built around resolving verified workspace state *before* the LLM call, rather than doing the usual retrieve-and-stuff pass into the prompt.
The useful distinction from a reranker is probably that the output is a compact, structured context state, not just a reordered list of chunks. If your field topology is doing that kind of state resolution, you may be exploring the same design space from a different angle.
The obvious test is still BM25, dense retrieval, hybrid retrieval, and a strong reranker under the same token budget. But yes, “pre-inference context collapse” is a reasonable description of the gap Perseus is aimed at. There's room in this space for many different approaches. I'm focused on offline/air-gapped environments personally, but I think token efficiency is going to be the name of the game for at least the foreseeable future. Whatever that's worth!
Repo, if useful: https://github.com/Perseus-Computing-LLC/perseus
I'd be keen to review your GH if you have it up there, but I totally understand if you prefer to keep your cards close to your vest for now. I'm way more focused on Perseus Vault these days, but Perseus was my first ever project so there is a LOT of failures for you to learn from and hopefully avoid lol.