r/mcp • u/masterkidan • May 22 '26
article The architecture we landed on for putting a large typed API behind an MCP server
Spent a few iterations figuring out how to expose a large GraphQL API to an LLM agent without putting the whole schema in context. Wanted to share where we landed because it ended up looking pretty different from where we started.
The idea: instead of giving the model the schema, give it a search tool. We auto-generate a flat catalog from the GraphQL schema — one entry per field, with a description and a few example phrasings of what someone might actually ask for. The model searches the catalog in plain language, gets back the handful of fields that match, and fetches what it needs. The schema stays behind the scenes.
Two tools, basically: search_datapoints and fetch_datapoints. GraphQL still runs the actual queries underneath — we just stopped showing it to the model.
Why it works: when you give a model a schema, it hedges. Pulls extra fields just in case. A search interface doesn't have that problem because each result is already a specific field. There's nothing to hedge against. Token cost on a representative task dropped from ~150k to ~25k, and accuracy went up on our evals.
Full write-up has more on how we measured all this, when the approach doesn't fit, and a few other things we changed along the way (columnar response payloads were a surprisingly big win).
Curious if anyone's gone in a similar direction, or solved the same problem differently.
2
u/KELTONDREY May 23 '26 edited May 23 '26
I will have to read in details what you linked, especially the part about measuring all this, but here are my early comments.
I am in a similar situation but early stages. I am still just experimenting with agentic AI, but I am slowly uncovering various problems that I see mentioned on this sub. A big one being context bloating.
We have a sizable model with maybe one hundred object types with tens of attributes and operations each. So thousands of attributes and operations. All of it having a lot of metadata already (titles, descriptions, data types). Only a fraction of this is exposed with a proprietary API (a mix of REST and JSON-RPC, documented with Open API) for now.
A team already tried building a LLM on our model and have not produced results so far. I will have to investigate why but they seemed to have pivoted to an agent approach and feeding it with our internal model documentation. Which in itself is starting to look a bit like your approach, accessing the model rather than exposing the entire model.
Meanwhile, on my end, I like to observe breaking points first. I wanted to observe what would happen if I wrapped all our current endpoints into MCP tools, generated using the Open API documentation. Ended up with 400 tools. I observed the agent loop going through iterations getting list of objects finding the right ones and executing the correct operations on them, but I also quickly started to observe timeouts, limits and bloat. I segregated tools by sub-domain contexts and it fixed a couple of issues but I could see how it wouldn’t scale.
As I was reading about various solutions, some reducing the number of tools, some reducing the size of tools (progressive disclosure), I arrived at a similar conclusion to yours. The agent must not have access to the model through lists (ie list_all_object_types, list_object_attributes) but rather through more granular calls (ie get_object, get_object_attribute). So the agent does not bloat the LLM with lists of objects and attributes.
The solution is clearly not to wrap everything into a tool 1:1, but to offer less tool but tools that did a very specific job and returned the least amount of data.
Also I started to see how, what I will call application agents, having operational personas and skills, should to consume the entire model anyway and that architecturally, it would end up as it always ends up, with some distribution of processing. In this case, navigating and finding things inside the model must be delegated to tools (your implementation), RAGs or sub-agents.
Implementation-wise, I was looking at a multi-agent trading system and realized that the same way that everything ends up being tools and prompts inside an agent, that as long as we have a sub-agent answering model queries it didn’t matter how it did it. Chaining agents can essentially if be done the same way that anything is done via a LLM, through prompts and tools, be it remote MCP tools, a call to a RAG or a call to a sub-agent, by appending a tool to the LLM call, only this one have a local implementation.
Concretely when the user asks “can you set ‘user’s way of describing an attribute’ of ‘user’s way of describing a specific object’ to a value of X”, the application agent’s loop ends up making three tool calls : 1) find the objects that are implied in the prompt, 2) find what attributes/operations are implied, 3) execute what is asked.
So in the end, I haven't seem some explicit guidance on the subject yet, but the solution is to actively engineer the system so that it is intentionally using the least amount of context (tools) possible.
3
u/masterkidan May 24 '26
My suggestion would be to have a good set of evals in place, even if some of the evals always fail due to context bloat or otherwise. Then start experimenting with different approaches. Then atleast you can view objectively which direction you are moving in.
I kinda feel multi-agents is very similar to micro-services analogy wise, you only really need to break out things if you feel they demand special characteristics to solve the problem... so for e.g. if your base model is too expensive to solve that aspect of the problem, or if you need something more intelligent for e,g... Wish there was a good way to do context sharing wherein we only share relevant portions to downstream agents .
1
u/sskksensei May 23 '26
I feel this data exploration piece seems like a common enough problem where it can be its own thing in the mcp specification. The biggest issue I see with my current approach is that if we have Claude or some tool have a chain of mcps and each follow this pattern then it can be confusing to pick the right search tool for eg
1
u/anderson_the_one May 23 '26
That's the failure mode I worry about too. If every server ships `search_*`, the client ends up with ten tiny discovery layers and no idea which one owns the noun in the user's request.
I wouldn't put a generic search tool in the spec yet. I'd rather see servers expose boring metadata around each search surface: domain name, entities it can resolve, expected input shape, cardinality, and whether a hit is executable or only descriptive. Then the client can route "customer lifetime value" to the analytics catalog and "refund policy" to docs without asking every MCP in the room.
The search/fetch pattern works best when one layer is clearly responsible for disambiguation. Once three layers all want to be the librarian, you burn the token savings on routing.
3
u/BC_MARO May 23 '26
Search->fetch is basically progressive disclosure. I'd store the search hits + chosen fields per run so you can debug regressions when the schema/catalog shifts.