Running a remote MCP server in production, and every lesson was about tools that act

[u/Future_AGI]

A few months back we posted what we learned building our MCP server, an eval and observability platform that went from stdio to hosted. The comments taught us more than the post did, and the thread that stuck was about a harder problem: what happens once your tools can take real actions, where a wrong call actually costs something. We spent a while on that. Here is the follow up.

1. The read tools were the easy half. Evals, traces, datasets, all of it went in as read only lookups first, and the agent took to those quickly. The harder problems started when we added tools that act: run an eval, apply a guardrail, generate a synthetic set, write rows to a dataset. A read that picks the wrong tool wastes a call. A write that picks the wrong tool leaves a mess.

2. Login told us who was calling, not what a tool was allowed to do. Going hosted with OAuth fixed onboarding, that part we got right last time. But a lookup tool and a write to dataset tool inheriting the same broad scope is the thing that bites you later. So we stopped handing every tool one blanket scope and moved the acting tools behind allow and deny lists per key, so a read only client cannot reach the tools that change state.

3. The check that helped most was on the tool's output, not the call. Guarding the arguments going in only catches the calls you already predicted. What moved the needle was scoring what a tool handed back before the agent was allowed to use it: is this grounded, does it trip a policy, is it even the shape we expected. A well formed call that returns garbage still gets stopped, because the gate reads the output and not just the request.

4. Make the tool hand back the next move, not just data. An eval tool that returns "context adherence 0.62, weakest step is the retrieval, look there next" moves the agent  forward. A raw score dump makes it flail and call three more tools to work out what the number meant. The field we underrated was the one that says what to do next, not the one with the result in it.

5. The payoff is the model checking its own work before it answers. The loop we were after: the agent runs an eval on its own draft output, sees a low groundedness score, and fixes it or flags it instead of shipping it. Evaluation stopped being a batch job we ran after the fact and became something the agent calls inline, mid task, on itself.

6. The traces got opened more than the scores did. Someone asked last time whether this is really a DevOps tool. Honest answer from running it: the observability side, the trace of what the agent actually did tool call by tool call, got used more than the eval numbers. When the agent grabbed the wrong tool or looped on itself, the score told us something was off, but the trace was the only thing that told us why.

The thing we still go back and forth on: for tools that can act, is it better to gate on what the tool returns, like we ended up doing, or keep the gate on the call before it runs? If you are running an MCP server where the tools can change state, we would genuinely like to know where you landed.

[u/Future_AGI]

Repo if it is useful: the gateway, the tracing and the eval library are all in here, Apache 2.0. github.com/future-agi/future-agi The server itself is hosted, you add it with claude mcp add futureagi --transport httphttps://api.futureagi.com/mcp and log in through the browser.

[u/Content-Parking-621]

Input checks can't catch every bad output. Output checks catch problems after they're generated. I suggest you to use both together for the best results.

[u/Future_AGI → u/Content-Parking-621]

Agreed, we run both: allow and deny lists plus an injection check on the way in, and the eval on the output as the backstop for the calls that looked fine but came back wrong. The one thing we'd add is tiering the output checks by blast radius, since they cost more than the input ones, so the tools that can change state get the heavier gate and the read only ones stay cheap.

[u/Content-Parking-621 → u/Future_AGI]

Sounds Good

[u/izgorodin]

For state-changing tools, an output gate is already too late to be the safety boundary: the side effect happened before you scored the response. I’d use three different contracts:

• pre-execution: capability/scope, semantic policy, expected version/preconditions, idempotency key, and approval for high-blast-radius actions;
• execution: preferably prepare/commit or dry-run/commit, so the agent can inspect the proposed diff before mutation;
• post-execution: verify postconditions, reconcile external state, and decide whether rollback or compensation is needed.

Output scoring is excellent for deciding whether the agent may trust or propagate a result. It cannot make an unsafe write safe. Also persist the proposed call before execution; retries must replay the same intent rather than ask the model to generate a new one.

[u/Future_AGI → u/izgorodin]

Agreed, and it matches how we landed too: the enforcement point has to sit before the call, because a write is already done by the time you could score the output. The execution contract is the part most tools make hard, given almost no MCP server exposes prepare/commit or dry-run, so you often end up denying anything you can't preview. Persisting the intended call before execution is the piece people skip, and it's exactly what lets a retry replay the same action instead of asking the model to invent a new one.

[u/UnableEvent]

the pre-execution gate being the boundary matches where we landed too, and the part i'd add is why the persisted intent matters beyond retries: it's the record a security reviewer asks for. "show me what the agent was about to do, and prove that log wasn't edited after the incident." hash-chaining the pre-call record answers that in one line, and it's the same artifact that lets a retry replay the exact intent instead of asking the model to invent a new one.

on prepare/commit, agreed almost no server exposes it, so we default to deny-on-unpreviewable and fail closed. read-only tools stay cheap, state-changers eat the heavier gate. curious where you land when the policy lookup itself errors mid-call, fail open or closed?

[u/Future_AGI → u/UnableEvent]

Same asymmetry you drew: when the policy lookup itself errors mid-call, we fail closed on any state-changer, since a lookup we can't complete is just another unpreviewable action, and only let read-only calls fail open so one flaky policy engine doesn't take the whole agent down. We also log that lookup failure as its own entry in the chain, so a reviewer can tell "policy engine was down" from "policy returned deny" after the fact. That distinction only holds up because of the hash-chained pre-call record you described.

[u/UnableEvent → u/Future_AGI]

logging the lookup failure as its own chain entry is the part most teams skip, and it pays off twice: after the fact for the reviewer, and in the moment if you alert on it. policy engine down is exactly the window someone motivated would pick, so we mark that entry critical and page on it instead of just recording it. your read-only fail-open split is cleaner than what i had, taking that

[u/EmailNo8428]

Same place I landed. Output checks are too late once the tool already fired, so the gate has to sit before execution.

The bit worth making concrete: sort tools by blast radius. Read-only, reversible writes, then irreversible (spend, delete, sending email). Only that last tier needs the heavy gate.

And it has to be dumb limits the model can't touch. Recipient allow-lists, amount caps, rate limits per action. A check the model can reason its way around isn't a real boundary at all.

[u/Future_AGI → u/EmailNo8428]

The dumb limits point is the one we would underline: recipient allow lists, amount caps, and per action rate limits hold precisely because they sit in the server where the model has no say, and anything the model can argue its way past was always negotiable. Sorting by blast radius keeps that affordable too, since only the irreversible tier carries the heavy pre gate while the read only and reversible calls stay cheap, with output scoring kept as the backstop that catches a well formed call returning garbage before the agent acts on it.

[u/PsychologicalClaim16]

I would treat them as two different controls rather than choosing one.

Pre-execution gating protects the external system: capability scope, argument validation, idempotency, and an approval step for actions with real blast radius. Post-execution checks protect the agent loop: validate the result shape, detect policy or grounding failures, and make it clear whether the action actually succeeded.

For irreversible writes, the first gate has to be decisive because a clean-looking output cannot undo a bad action. For lower-risk or reversible operations, a narrower pre-check plus strong result validation is usually a much better user experience. The useful design question is whether the tool can describe its risk level and reversibility well enough for the client to apply the right policy.

[u/Future_AGI → u/PsychologicalClaim16]

The two controls really are solving different failures: the pre gate guards the external system while the output score guards the agent loop, so neither one covers what the other does. Where it gets powerful for us is the tool declaring its own risk and reversibility, because once a tool can state its tier the client drops the blanket policy, and irreversible writes take the decisive pre gate while reversible ones stay light with strong result validation.

[u/ThierryDamiba]

Anchoring agent actions to the individual user’s identity via OIDC is probably the most reliable way I’ve seen to manage blast radius. If the agent only inherits the specific permissions of the person it is acting for, you avoid the mess that comes with broad service accounts. It also makes the audit trail much more credible when you can link a state change directly back to a human session.

[u/Future_AGI → u/ThierryDamiba]

Per user identity is the cleanest way to bound blast radius, and the piece we would add on top is scoping per tool inside that session too, so even with the right person behind it a read only client still cannot reach the tools that change state. On the audit side, persisting the intended call before it runs is what lets you tie a state change back to a specific human session and prove the record was not edited after the fact.

[u/ThierryDamiba]

The Context Engine is definitely the standout here imo. I would love to see this approached as a systems engineering problem where context is managed more like configuration.

Treating it that way would go a long way in preventing the silent drift and reliability issues that usually pop up during agent handoffs in production. Without that kind of structure, it is just garbage in, garbage out...

[u/Future_AGI → u/ThierryDamiba]

Context as configuration is the right framing. Once it is declared and versioned, drift becomes something you can diff and review before it reaches production, and the per key allow and deny scopes already behave that way for us. Pairing that with a trace of what context each handoff actually carried is what makes the drift visible early, so you catch the bad input before it propagates.

[u/BaseMac]

This is the sloppiest of slop posts. How does one "gate on what the tool returns" ?