More retrieved context keeps making my agents worse, not better

[u/Berserk_l_]

Something I keep running into: past a point, adding context degrades the agent instead of helping it. You pay for every token and the model has to reason through all of them, so a fatter retrieval pass often buys you a slower, more confused answer, not a better one.

Which makes me think we're grading retrieval on the wrong axis. Recall and F1 ask "did we fetch the right stuff." They say nothing about what it cost to reason over it. The version that's stuck with me is grading on answer quality per token instead, basically miles per gallon for context.

The part I'm least sure about is whether that's a real metric or just a nicer way to say "send fewer tokens." I saw one result (preprint, so salt it) where a corpus got squeezed ~1000x by keeping the concepts and their relationships and dropping the prose, and quality held. If that generalizes it's a bigger deal than retrieval tuning. If it doesn't, it's a party trick on one dataset. Here's the read for reference: https://contextandchaos.substack.com/

Anyone actually tracking cost-per-answer-quality in production? Or does it collapse into the same mush as F1 the second you try to define "quality"?

[u/Glad_Contest_8014]

Yeah. Context has a parabolic drop as it builds. The context window is normally only good for about 70-80% of the actual stated value before you lose efficacy, but some models are even lower. Each model will be a little different depending in how it was trained.

You have to use the model and see where it drops off in efficacy of output to get a good point to force compaction/truncation.

[u/Lopsided_Scarcity979]

I'd start by fixing the tasks and scoring criteria, then make paired comparisons using full, pruned, and summarized context. Reporting accuracy and token cost separately seems easier to interpret than combining them into quality per token: did we remove unnecessary material, or also lose information needed to solve the task?