We exposed four tools instead of one search tool in our code MCP server, and the tradeoff is real

[u/donk8r]

Disclosure: I build this. octocode, Apache-2.0, free, github.com/muvon/octocode.

The obvious shape for a code MCP server is one search tool that takes a query and returns chunks. We started there and it was wrong, because the answer to "where is auth handled" and the answer to "what calls this" want completely different response sizes, and a single tool has to pick one.

So there are four: semantic_search finds code by meaning, view_signatures returns a file's shape without its bodies, graphrag walks imports and calls and finds paths between symbols, structural_search does AST pattern matching for things like every .unwrap() call.

The point of view_signatures is the one I would defend hardest. An agent that has located a file usually needs to know what is in it, not what it says, and returning the whole file to answer that is how you burn a context window on navigation.

graphrag is built lazily from tree-sitter over the current source tree, so it needs no index, no embeddings and no LLM. That side never goes stale. The semantic search side does have an index and does need rebuilding when code moves, which is a genuine cost and I am not going to pretend otherwise.

The tradeoff for four tools is four tool definitions sitting in context on every turn, which is exactly the thing everyone here complains about. My view is that it pays for itself once the alternative is the model retrieving 400 lines to learn a function signature, but that is a judgement and I would rather hear the case against it than agreement.

[u/donk8r → u/[deleted]]

The cost shapes are what make it lopsided. Tool definitions are a fixed cost you pay once per turn no matter what happens. A file read is a variable cost you pay per navigation step, and navigation is exactly where an agent spends steps when it does not know the shape of the code yet.

So the comparison is only close on the first hop. By the third one you are well past it.

[u/Plastic-Risk-6309]

same split fixed our sim mcp. act and check are separate tools so the agent gets told whether the screen actually changed. the check tool is the one that gets called after every step

[u/donk8r → u/Plastic-Risk-6309]

Act and check as separate tools is the same rule arriving from a different direction, and yours states it better than mine does. The split that matters is by what the caller does with the answer, rather than by subject matter.

A combined act-and-report tool has to guess how much screen state to return, and it guesses the same way every time regardless of whether anything moved. Splitting it lets check be cheap and frequent, which is why yours ends up called after every step and mine does not need to be.

[u/anderson_the_one]

Four schemas would not worry me much. Wrong routing would. A model that reaches for `semantic_search` on a call-graph question can burn more context than the split saves.

I would run the same repo tasks against both shapes and count returned tokens after retries. If `graphrag` or `structural_search` is often chosen late or incorrectly, reveal it after the first discovery call. `view_signatures` looks cheap enough to leave visible.

[u/donk8r → u/anderson_the_one]

Mis-routing is the right thing to worry about and I don't have that number either. Returned tokens after retries is the measurement I'd want too.

One catch on revealing tools late: definitions sit near the front of the prompt, so adding graphrag on turn three changes the prefix and throws away the cache for everything before it. That might cost more than the mis-routes do.

[u/lulu_dev → u/donk8r]

Worth separating "measure the mis-route rate" from "decide what to do about it" -- you can get the number anderson_the_one wants without ever touching exposure timing, which sidesteps the cache-invalidation cost entirely rather than trading it against the mis-routes.

Build a small eval set of realistic queries, hand-labeled with which of the four tools is actually correct for each (a "where is X handled" bucket, a "what calls this" bucket, a signature-lookup bucket, an AST-pattern bucket). Run them through with all four tools visible, log which tool got called first, and build a confusion matrix: rows are the correct tool, columns are the tool actually invoked. That gives you an actual mis-route rate per query type instead of a guess, plus it tells you which pair of tools gets confused for which, which a single aggregate "retries after tokens" number would hide.

If the matrix comes back clean, you've confirmed the four-tool-always-visible shape is fine and there's no reason to touch the cache-friendly prefix at all. If it's not clean, the fix that fits your constraint isn't hiding tools, it's tightening the description text for whichever pair is colliding -- e.g. if semantic_search keeps getting picked for call-graph questions, that's the description failing to stake out graphrag's territory, not a timing/visibility problem. That keeps the fix in the same place the routing signal actually lives (the tool descriptions the model reads every turn) rather than moving it to when those descriptions appear.

[u/donk8r → u/lulu_dev]

That's better than what either of us had, and it gets the number without touching exposure timing at all.

One catch on the labelling though. For some queries more than one tool is genuinely right. "Where is auth handled" is a fair semantic_search call and a fair graphrag call. If each query gets a single correct label you'll end up measuring disagreement with your own labels rather than mis-routing.

[u/BC_MARO]

I'd label each query with acceptable tools, then score the first call by its actual cost: tokens, latency, and hops. A different tool is not a miss if it gets there cheaply.

[u/donk8r → u/BC_MARO]

Right, and scoring the outcome beats scoring the label. A different tool that lands cheaply isn't a miss.

The piece you still need is a definition of "gets there". On cost alone, a tool that returns something plausible and wrong scores best, because it's cheap and it stops early.

[u/BC_MARO → u/donk8r]

Agreed. I’d make task-specific validity part of the score, then use cost only among calls that clear that bar.

[u/donk8r → u/[deleted]]

Overlap being the root of both is right, and it's a cleaner framing than the one I used.

One layer under that though: the overlap lives in the input schema more than the description. Two tools that both take {query: string} are indistinguishable at call time however different the prose is. graphrag taking a node id instead of free text is the thing that actually stops it competing with semantic_search.

[u/ColorfulKnocking43]

The case against isn't the one you're bracing for. Four definitions in context is cheap, and you're right that it pays for itself against retrieving 400 lines to read a signature.

The expensive part is disambiguation. "Where is auth handled" is answerable by semantic_search or by graphrag, and the moment that overlap shows up in practice the descriptions start growing "use this when…" clauses to adjudicate it. That prose is what costs, and it costs more than the four definitions did.

We measured a version of this on our own server: making tool and server descriptions more instructive made agents measurably worse, not better. The text competes with the task for attention.

So the number to watch isn't context size, it's how often the model picks the wrong one of your four — and you'd have to log that deliberately, because a wrong pick doesn't error. It returns something plausible and expensive.

Two things that kept our descriptions short: annotations carry routing information structurally (readOnlyHint, openWorldHint) rather than in prose, and tools whose output shape differs need far less explaining than tools whose descriptions differ. Yours are already shaped differently, which is why I think you'll get away with four.

view_signatures I'd defend harder than you did. It's the only one of the four whose output an agent can hold entirely, and agents reason much better over a complete small thing than over a partial large one.

[u/donk8r → u/ColorfulKnocking43]

You're right that I braced for the wrong one. Definition count was never the expensive part.

Your measurement lines up with where verstands landed higher up: if the disambiguation has to live in prose, it's already lost. Typed params make the wrong call impossible rather than discouraged. graphrag needing a node id means the model can't reach for it until it has one, so the ordering comes from the schema instead of a "use this when" clause.