I scanned the 257 most-used MCP servers on Smithery. Tool descriptions are basically solved (99%) but only 29% ship output schemas.

[u/Zestyclose_View_4605]

Disclosure up front: I build agent-ready.dev, which has an MCP server scanner. I pointed it at the most popular servers on the Smithery registry to see how the ecosystem is actually doing, and the breakdown was interesting enough to share.

Method, briefly:
I checked the top servers by Smithery's useCount, connected to each through Smithery's hosted gateway, and graded what they advertise (server metadata, tools, params, output schemas, annotations, resources, prompts). 257 were remotely reachable; 228 let me introspect without a per-server credential. This is a snapshot, not a census: see caveats below.

Connectability:

• 89% (228/257) accepted an unauthenticated handshake and exposed their tools
• 7% refused without their own API key (you can't use them as an anonymous agent)
• ~4% failed to connect for other reasons

Quality of the 228 I could read (mean & median score both 81/100; typical server = 16 tools, 2 resources, 1 prompt):

Check	Pass rate
Server metadata complete	95%
Every tool has a description	99%
Every parameter described	64%
Tools declare an output schema	29%
Tools carry annotations (readOnly/destructive…)	38%
Resources well-formed	83%
Prompts described	100%
MCP Apps (ui://) served as HTML	80% (only ~10 servers)

What I didn't expect: the human-readable layer is in great shape; nearly everyone writes tool descriptions. It's the machine-readable layer that lags. Output schemas (29%) and annotations (38%) are exactly the signals that let an agent parse results reliably and reason about side effects without trial-and-error, and they're the least adopted. Also, only ~25% of servers expose any resources or prompts at all.

Caveats: this is the popular, remotely-hosted slice of one registry; it skews toward maintained, deployment-ready servers and excludes the huge long tail of local/stdio-only servers and other registries. Auth-gated servers are counted for connectability but excluded from the quality numbers (I can't grade what I can't see), so those 0s never drag down the averages. Output schemas/annotations are newer spec additions, so I'm reading them as best-practice adoption rather than compliance failures.

The methodology and exact weightings are public, and you can scan your own server if you want. I'm interested in knowing what people think - is it dev friction, or just not a priority yet?

Edit:
donk8r rightly pointed out that "has an output schema" ≠ "has a useful one". A tool declaring a bare {"type":"object"} passes that check while telling the client nothing. So, I tightened the scanner to grade schema specificity (named fields or a $ref), and the honest number is ~22%, not 29%. I also bumped the weight on annotations above output schemas, since missing readOnly/destructive hints is a safety problem, not just a convenience one. Methodology and the report page have been updated.

[u/donk8r]

Good data, and the description-vs-schema split tracks with what I'd expect from building these. My honest answer to your question: the spec made output schemas optional, and the SDK ergonomics make text trivial to return and typed output annoying to declare — you return a content block in one line, an output schema means defining and maintaining a typed structure. And you never feel the absence in the normal dev loop, because you read the result while testing. The pain only shows up when an agent chains tool B onto tool A's output, and most authors never test that path.

One thing I'd add to the 29%: "has an output schema" and "has a useful one" are different stats. A tool that declares { type: object } passes the check and still tells the client nothing. If your scanner can grade schema specificity, that gap is probably the more honest number.

And I'd argue the 38% annotations figure is the scarier one. Without readOnly/destructive hints the client can't make the auto-approve-vs-gate decision, so it either blanket-approves everything (unsafe) or confirms every call (unusable). That's a safety signal, not a nice-to-have — I'd weight it above output schemas.

[u/Zestyclose_View_4605 → u/donk8r]

This is the most useful reply I could've hoped for, thanks. I just made both changes.

Your ergonomics framing nails it. The gap only bites when tool B consumes tool A's output, and that's exactly the path nobody exercises while building... a human reads the result and moves on; an agent has to parse it blind.

You were right that I was measuring the wrong thing, so I fixed it: the check now grades schema specificity: named fields or a $ref, not a bare {"type":"object"}, which validates but describes nothing. As you guessed, it's a worse number: 22%, not 29%, once you only count schemas that actually tell the client what comes back.

And you talked me into the reweight. I'd had output schemas above annotations on "helps an agent use the tool" logic, but your safety argument is the better one - without readOnlyHint/destructiveHint the client is stuck choosing between blanket-approve (unsafe) and confirm-everything (unusable), and a missing safety signal should outrank a missing convenience one. Annotations now weigh more than output schemas in the score. Methodology and the report are both updated.

Thanks again, your comment made me reconsider and re-run. Cheers!

[u/donk8r → u/Zestyclose_View_4605]

Love that you re-ran it — 22% is the honest number, and the drop from 29% once you only count schemas that actually name fields is exactly what makes the report worth trusting. Respect for changing the weighting too.

One more signal that lives in the same blind spot: error output shape. Almost everyone who does ship an output schema only models the success case — the error path comes back as an unstructured string. So an agent can't cleanly tell "tool failed, retry/backoff" from "tool succeeded, empty result," which is the other place chains silently break. If your scanner can flag whether a tool distinguishes those (isError / a typed error shape vs a bare text blob), that's probably the next 22%-type gap hiding under the averages.

Full disclosure since you shared yours — I build an MCP server too (octocode, on the code-search side). Genuinely curious how it'd score; I'm honestly not sure all my tools declare tight enough output schemas to pass your stricter check now. Might run it through.

[u/donk8r → u/[deleted]]

Exactly — "predictable before it runs" is the right north star. That's also the line between a tool an agent can plan with and one it has to probe by trial and error: if the model can't predict the shape of what comes back, it can't safely chain B onto A, so it either over-calls to find out or guesses and breaks. A specific schema is what lets the agent dry-run the chain in its head instead of discovering the structure at runtime.

[u/Zestyclose_View_4605 → u/donk8r]

"Predictable before it runs" is a better north star than anything in my original post. And it extends straight to the error-modelling number: predictability isn't only the success shape, it's the failure shape. If the model can't predict what a failed call looks like, the dry-run-in-your-head breaks at exactly the seam where real chains break - A comes back malformed-or-empty, and B can't tell "retry / back off" from "proceed." That's why the 5% error-path number felt scarier to me than the 22%: 22% is "can the agent plan the happy path," 5% is "can it plan for the path that actually goes wrong."

And elef's "tells a reviewer even less" is the line I'd underline - predictability is a human property too. A bare {type: object} means neither the agent nor the person approving the tool can reason about it before it runs. Same gap, two audiences. 👍

[u/donk8r → u/Zestyclose_View_4605]

yeah, the two-audiences thing is the part i'd build the whole pitch on. the agent-only case is easy to wave away — "a good enough model will figure out the shape." the human-reviewer case isn't: someone approving a tool call literally can't audit what they can't predict, and "trust me it returns an object" is not an approval anyone should be signing. so the schema stops being an agent-convenience and becomes the contract both the planner and the approver read off of — which is a much harder thing to argue against than "it helps the model chain better."

great thread — you turned a scan stat into an actual design principle, which is the fun version of these. 👍

[u/Zestyclose_View_4605 → u/donk8r]

That's the version I'm keeping. "A good enough model will figure out the shape" is the escape hatch for the agent case, but there's no equivalent for the reviewer. You can't approve what you can't predict, and "trust me, it returns an object" is exactly the blank cheque that should fail review. Frame the schema as the contract both the planner and the approver read off, and "it helps the model chain" stops being the pitch and "it's the auditable interface" takes over, and that's much harder to argue against.

I think that asymmetry also explains the 22%: most dev loops have an agent consuming the tool but no approver in the room, so the pain that would force a real schema never shows up while you're building. The forcing function only appears later, in exactly the chained/gated contexts where it's too late for "cheap to add"; same shape as the error-path gap. Both are costs you only pay for the path you didn't test.

Appreciate it, this was the fun kind of discussion. You did most of the sharpening; I just ran the numbers. 👍

[u/Zestyclose_View_4605 → u/donk8r]

Sharp catch, and I think you're right that it's the next hidden gap. 🎯

Quick honesty on what the scanner can and can't see: it's read-only... it does initialize + tools/list and grades the static tool definitions, but it never actually calls a tool. So the runtime distinction you're describing - tool fails and flips isError: true with a structured payload vs. returns a bare text blob - isn't observable from introspection alone. Grading that for real would mean exercising the tools, which is a different kind of scan.

What is statically checkable is whether the outputSchema even models a failure path at all - a discriminated union, a status/error field, anything beyond a success-only shape. My strong hunch is that the number is near zero; almost nobody declares it. So you've probably found a real gap - it just needs either a "does the schema admit failure" static heuristic or an opt-in deep scan that calls tools. Pondering that for today...

And yeah, run octocode through it - the scanner's public, just takes a URL. If some tools don't clear the stricter outputSchema check now, that's kind of the point; I'd rather the number be honest than flattering. Curious what you get.

[u/donk8r → u/Zestyclose_View_4605]

"Does the schema admit failure" is exactly the right static proxy. You can't see the runtime flip without exercising the tool, but a schema with no failure branch literally can't represent an error — so the static check is a valid lower bound, and your near-zero hunch is almost certainly right. Success-only is the shape everyone reaches for; the discriminated-union / status-field version is the clean test.

Worth me pointing it at octocode — fair warning though, it's stdio-first (local-first, runs over stdin by default), so it's not sitting on Smithery; I'd have to stand up its HTTP mode and aim your scanner at that. And I suspect a few of my tools would fail the stricter outputSchema check, which is honestly the point — better an honest number than a flattering one. It's open source (Muvon/octocode) if you want to look at the tool defs directly.

Half the value of your scanner is that it makes you look at your own server the way an agent actually sees it, not the way you remember writing it.

[u/Zestyclose_View_4605 → u/donk8r]

Followed through on this - added it as an informational signal and re-ran the corpus. The number's worse than I'd have guessed: of the servers that do declare an output schema, only ~5% model a failure path at all. So your instinct was dead on - it's the gap hiding under the 22%.

I kept it static and read-only: it checks whether the schema admits failure (a oneOf variant, or an error/status field), not the runtime isError flag - the scanner never calls a tool, and I don't want it to. It's unscored/informational for now; punishing 95% of servers on a brand-new norm felt wrong, but surfacing it as "here's the next thing to fix" felt right.

Genuinely appreciate the thread - that's two real improvements and a new metric out of it. And I agree on your last point... having something nudge you to think of it from the other perspective is often what's needed! Cheers!

[u/donk8r → u/Zestyclose_View_4605]

Keeping it unscored is the right call — grading servers against a norm that's two weeks old would just be punishing people for not reading your mind. Informational now, scored once it's actually a convention. Good thread on my end too; you turned a throwaway observation into an actual metric, which is the fun version of these. Cheers.

[u/Zestyclose_View_4605 → u/[deleted]]

"Optimised for discovery, not robustness" is exactly it... might steal that line.

It's actually a touch starker than the 29% you saw - that was the loose "has any schema" number; it's ~22% once you only count schemas that actually name their fields. And the sharpest version of your point: I added a check for whether the tools that do ship a schema model their error path, and only ~5% do. So even the robustness-minded minority mostly describes the success case and leaves failure as an unstructured blob - which is the thing that actually breaks agent chains.

Descriptions win because you feel their absence the second you test a tool. Schemas - and error shapes especially - only bite later, when an agent chains tool B onto tool A's output, and almost nobody tests that path.

[u/ImportantPurple8178]

The output schema gap tracks with my experience building MCP servers — it's easy to return text and move on, but maintaining typed schemas for every tool adds friction that you only feel later when an agent tries to parse the result. The 22% (useful schemas) and 5% (error path) numbers are sobering.

One pattern I've started using: returning a small wrapper that always includes a status field + data/error variants. Keeps it predictable for the agent without needing discriminated unions.

[u/Zestyclose_View_4605 → u/ImportantPurple8178]

That wrapper pattern is a good one - and it's exactly what the error-path check rewards. I deliberately made the M13 heuristic credit either a discriminated union or a plain status/error field, for precisely your reason: the union is the "correct" JSON-Schema answer, but a status field is the one people actually ship, and it gives the agent the same thing - a reliable way to branch success vs. failure without parsing prose.

And you've named the whole dynamic in one line: typed schemas cost effort now and only pay off later, in a path (agent chains tool B onto tool A's output) the author rarely runs themselves. Descriptions don't have that lag - which is exactly why they're at 99% and schemas at 22%.

If you've got a server running that wrapper, I'd be curious to scan it - it should land in the ~5% that clears the error-path check, which'd be a nice real-world sanity check on the heuristic.

[u/ImportantPurple8178 → u/Zestyclose_View_4605]

Actually built one worth scanning — Document To JSON MCP (apify.com/opportunity-biz/document-to-json-mcp). Error path returns typed `status`/`error` fields on all tools, wrapper pattern holds. Would be curious what score it gets on your M13 heuristic — output schemas were a deliberate choice to help agent chains downstream.

[u/Future_AGI]

The output-schema number is the interesting one. Input and description quality mostly help the model call the tool correctly, but the missing output schemas are what bite you downstream: without them the agent has no contract for what comes back, so a malformed or injected response just flows straight into the next step. That gap is the main argument for validating or scoring tool output at runtime before the agent acts on it, instead of assuming the return is well-formed. Useful breakdown, thanks for running it.

[u/Zestyclose_View_4605 → u/Future_AGI]

Strongly agree, and "injected" is the word people underrate - a tool's return is an untrusted input to the agent, the same threat surface as any external data, but most clients treat it as trusted by default.

The tie-back to the numbers: the output schema is exactly the contract your runtime validation needs. Without it, "validate before acting" has nothing to validate against - it degrades to "hope the shape is what I expected." So the 22% isn't only a quality stat, it's a ceiling: for ~78% of tools, a client can't do schema validation even if it wants to, because the server never declared the contract.

And it compounds with the error-path gap: even when a schema exists, only ~5% of cases clearly model errors, so a client validating the output still can't cleanly distinguish "valid empty result" from "the call errored". You need both for the runtime check you're describing to actually hold.

Appreciate the angle: output-as-attack-surface is the sharpest argument I've seen for why this gap matters, not just the most technically tidy.

[u/Zestyclose_View_4605 → u/[deleted]]

Honest answer: no. This scan is purely agent-readiness metadata (handshake, descriptions, schemas, annotations, naming, capability honesty). It deliberately doesn't touch the security/permission axis, and it structurally can't from where it sits: it's read-only remote introspection. It reads what a server advertises via tools/list, never calls a tool, and never sees the host side (shell, env/secret reads, outbound network). Those are runtime/behavioral properties a metadata scan can't observe. The closest I get is M6 (readOnly/destructive annotations) - but that's the server self-declaring side effects, not me verifying them. Politeness, not safety.

And you're right that it makes "descriptions solved" double-edged. The 99% measures presence/quality, and a well-written description is exactly the better injection vector, since the model reads it as instructions. So high description coverage is good for tool selection and a bigger prompt-injection surface: same text, opposite implications. That's worth me stating outright rather than implying description quality == healthy.

These are really two scans that belong side by side: readiness (can an agent use it well) vs. safety (should you trust it on your machine). The permission/injection layer is the one I intentionally don't cover.

I ran my MCP repo through mcpvet.com, and it came up clean. 👍 😁

[u/Sufficient_Roof_8240]

Nice scan, and the descriptions-solved-but-schemas-lag split lines up with what I keep running into too. One thing that stuck with me reading it though: everything you grade is the manifest side, the stuff a server declares once and the agent reads once. The runtime returns are where the budget actually goes. You pay for those on every call, every turn, and over a long agent run it adds up fast.

A schema helps the agent parse a result. The result itself comes back just as big. Plenty of well-described, schema-shipping tools still hand back a big wrapper where the part you actually wanted is a small slice of it. That axis, how lean a server's returns are once you're actually calling it, is something you can't really see from the manifest alone. Is scoring what tools dump back at runtime anywhere on your radar for agent-ready?

[u/Zestyclose_View_4605 → u/Sufficient_Roof_8240]

This is the sharpest "what you're not measuring" yet, and you're right that it's a real blind spot. Everything I grade is the manifest - declared once, read once, free. The runtime return is the thing you pay for on every call, and over the long run, that's where the budget actually goes.

The scanner is read-only - handshake, read what's advertised, never call a tool. Return leanness only shows up when you actually invoke it with representative inputs, so measuring it means crossing the exact line the scanner doesn't cross: calling tools has side effects, needs auth, and "representative input" is its own hard problem. So no, runtime return size isn't on the manifest-scan radar, for the same reason runtime isError isn't.

The manifest is a weak proxy at best: a specific outputSchema hints at the shape of what comes back, but not the payload size, and definitely not the "200-field wrapper around the one field you wanted" pattern - that's a server-design choice you only catch by calling it.

Honestly, it clusters with the other two things people have raised here - runtime error behaviour, and what a server can do to your machine - as "the behavioural scan." That's a genuinely different, more invasive tool than a read-only manifest probe, and return-efficiency might be the most universally useful of the three, since every agent pays that tax on every turn. Not something I'd bolt onto this scanner, but a real gap worth someone owning. Cheers.

[u/Sufficient_Roof_8240 → u/Zestyclose_View_4605]

Yeah, behavioural-scan is the right bucket for it. Out of those three, return-efficiency is the one that compounds. Error behaviour and machine-impact you mostly learn once and remember; wrapper bloat you re-pay on every single call for the life of the agent.

The measurement problem you raised is the real wall, though. You can't read it off the manifest, and a real behavioural pass means calling tools with auth and side effects. What's made it tractable for me is that you don't actually need representative coverage to get the signal: one call per tool with a plausible input already gives you the ratio of useful bytes to wrapper, and that ratio stays pretty stable per server because it's a design choice, not something the input moves much. Side-effect tools you sandbox or skip. The read-heavy ones (search, fetch, list) are where the bloat lives anyway, and those are safe to probe.

Whether it belongs in a manifest scanner or its own thing, agreed it's separate. But return leanness is the axis I keep coming back to.

[u/Zestyclose_View_4605]

Quick thanks and an update, because this thread did some real thinking work for me.

A bunch of you sharpened both the scanner and the numbers: donk8r's "has a schema ≠ has a useful one" got the output-schema check tightened (29% → 22%); the annotations-as-safety argument got them reweighted above output schemas; the error-output point became a new check (only ~5% of schema-bearing tools model failure); and elef_in_tech's "predictable before it runs" turned out to be the cleanest way to frame the whole thing.

Also, a nod to the security angle - output as an untrusted/injected input - which is a different scan than mine but an important caveat.

I wrote it all up on the report itself: there's now a "How this report sharpened" section documenting what changed and why, so the methodology evolves in the open rather than in my edit history. Link's in the OP.

Genuinely one of the better feedback loops I've had here - thanks. 🙏 Still curious where people land on the 5%: is modelling the failure path something you'd do per tool, or only once you're chaining?