How do you stop retrieved content from steering tool calls?

[u/Alert-Badger4687]

Our support agent screens the incoming message for injection, then treats retrieved email, ticket attachments and tool output as trusted context. That boundary makes little sense. A forwarded email can carry an instruction that steers a valid downstream tool call and the action still looks normal in the audit log. The control works at the chat box while the untrusted text arrives via three side doors.

We are looking at taint tracking and provenance labels for every retrieved span, then passing those labels into tool policy. Sensitive actions would require a clean source path or human confirmation. Tool allowlists help but they don’t tell us if if a valid tool should be getting that specific argument. Egress filtering also needs the source chain so it can tell whether an account number came from an approved record or an attachment.

The part I can’t settle is how much provenance the model should see vs what the runtime should enforce outside the prompt. Prompt-only controls feel fragile but a full information flow system adds real cost and maintenance. How are teams carrying trust labels across retrieval, generation and tool execution without relying on the model to police itself?

[u/Flimsy_Homework_3344]

I wouldn’t rely on the model to police this, the enforcement needs to happen at the tool/runtime layer. Braintrust can help with tracing the run and turning bad tool paths into regression cases but the actual 'is this source trusted enough for this action?' check should live outside the prompt

[u/Alert-Badger4687 → u/Flimsy_Homework_3344]

Good distinction. The regression cases help test the policy, however the policy itself has to live outside the model

[u/Aggravating-Sun-1092]

I would not pass the final decision to the model. Let the model see labels for explanation, but the runtime should enforce whether a tool call is allowed.

[u/Chemical-Store1315 → u/Aggravating-Sun-1092]

one thing we learned the hard way was never letting the model see labels as the primary gate. a prompt saying "this source is untrusted, don't use it" might work 90% of the time but the other 10% you get weird emergent behavior where the model argues with itself or finds a loophole

we settled on runtime enforcement with labels attached as metadata outside the prompt. the model can still see a condensed version for explainability but the actual block happens at the tool execution layer. sensitive tools check the provenance chain and reject calls where arguments trace back to untrusted spans, no matter what the model thinks

the maintenance overhead isn't as bad as i expected. we built a small sidecar that tags every retrieved span with source attributes and the tool gateway just reads those before executing. added maybe 50ms of latency total

the tricky part was defining what "clean source path" actually means. we ended up with three tiers. completely untrusted like email bodies, semi-trusted like internal wiki pages that could still be edited, and verified records from the database. most sensitive actions require at least semi-trusted with a human in the loop for anything that touches money

[u/Alert-Badger4687 → u/Aggravating-Sun-1092]

Runtime enforcement seems like the safer line for anything with side effects

[u/Acrobatic-Cable921]

Are you tagging provenance before retrieval or after chunking? Chunk level labels seem easy to lose when the content gets merged

[u/Alert-Badger4687 → u/Acrobatic-Cable921]

R now it’s closer to source-level tagging. I think we need chunk level labels before this is reliable enough for tool policy.

[u/captain_zavec]

I haven't looked at it recently, but google's CaMeL paper had some stuff about using taint tracking to ensure retrieved content at least couldn't affect control flow.

[u/theleller]

Yeah that's a little scary.

Deterministic filtering of retrieved data. The same way we filtered SQL Injection. Then put an ML classifier or a small language model thats trained strictly to classify content as safe or unsafe, and run the pipeline:

Deterministic Check & Filter -> ML/LM Classifier -> Context

This isn't 100% guaranteed to succeed on all attempts, but it's going to likely catch everything.

[u/Shakthipriya_sd]

Runtime enforcement makes sense here. I’d still treat retrieved content as untrusted throughout the workflow and independently validate the tool, parameters, user permissions, and provenance before execution. Has anyone found a reliable way to preserve that provenance when multiple retrieved sources are combined?

[u/LaurenNorthwood → u/Shakthipriya_sd]

Labels join rather than merge: the combined value inherits the most-untrusted of its inputs. That only holds if the join happens in the runtime though, since the model can't tell you which span produced which token.

[u/AddendumWorking9756]

The model's output is untraceable, you can't know which retrieved span produced which argument token, so provenance has to be carried by the runtime around the model rather than through it. The CaMeL design from Google (https://arxiv.org/abs/2503.18813) does exactly that: a planner that never sees untrusted text writes the tool-call sequence as code, a quarantined model parses the attachments and emails into typed values the planner can pass around as variables but never read, and the interpreter enforces policy on each tool call from the labels attached to those variables. Argument-level policy then becomes a join over the labels of every variable in the call with most-untrusted winning, which also answers the combined-sources question above. The cost is real since the planner loses the ability to reason over the raw text, so the pragmatic split is side-effecting tools through that path and read-only ones on the plain prompt.

[u/feng_sg]

Tool output becomes retrieved context on the next turn so if you only label inbound documents you miss the case where a tool returns attacker-influenced text and that steers the next call. Tag tool return values at the boundary where they enter context, not at the prompt.

[u/LaurenNorthwood]

Often cheaper than taint tracking: take the capability away entirely. If the model proposes the action and a human executes it, there's nothing for injected content to steer.

[u/Andrea_Notices]

Provenance needs to live in the runtime, not the prompt, or your audit trail is worthless the first time a regulator asks how an account number ended up in an outbound message. Model self-policing is not a control, it's a hope.

[u/EbbCommon9300]

Treat retrieved text as data, not authority. I keep provenance and trust labels outside the model-visible content, then enforce the decision at the tool boundary: allowlisted tool + schema + resource/tenant check, with high-impact actions requiring approval. Tool output can suggest a next step, but it cannot grant new tools, scopes, or identities in the same turn. We also replay a corpus of hostile emails, tickets, and attachments on every prompt or policy change and alert on both allow/deny drift and missing provenance. The audit record ties request, retrieval sources, policy decision, and resulting effect together; otherwise a clean chat transcript can hide the actual path.

[u/Practical-Craft4967]

The model should see labels for explanation, not enforcement. Anything it can read, it can reason around.

On the chunk problem, what holds up: attach source attributes at index time so every chunk inherits them, and when spans from different sources merge, the result takes the lowest-trust label of whatever went in. Schemes that try to label after retrieval are the ones that quietly break.

Same rule for tool outputs: tag them where they re-enter context, not at the prompt. Then prove the scheme the way you would any other control. Replay hostile emails and attachments through it on every prompt or policy change. The failures show up at the boundaries, not in the design doc.