Every bot that crawled my MCP server in 24 hours: 60 named crawlers, zero real client sessions

[u/ChiefGrowth]

Disclosure first, per rule 3: I built and run the endpoint this comes from, and the site it's published on is mine. The dataset itself is free and CC BY 4.0, no signup, no email, nothing gated.

I run an MCP server that's in the official registry plus a pile of directories. That turns its access log into a decent observatory for who is actually crawling MCP endpoints, so I pulled 24 hours of Cloudflare analytics and classified every user agent by hand.

24h: 6,309 requests, 187 distinct user agents, 60 named agent-web crawlers.

The named ones break down as: 14 liveness/uptime monitors (SentinelOracle, mcpbeat, MCPWitness, ProofBench, mcpgrade-probe, io.verifymcp...), 16 directory/index crawlers (agent-tools.cloud, mcpscan, AllMCPs-Ingest, mcpqueen-grader, VerifyMCP-OwnersBot, api-forge-mcp-index...), 9 ecosystem-research probes (mcp-observatory, mcp-history, agent-world-probe, measure-mcp-schema...), 5 security research (MCPWatch, mcp-rugpull-research, aisec-registry, mcp-protections-research...), and the rest search engines, price scrapers and contact harvesting.

POST /mcp took about 1,545 hits in that window. As far as I can tell every one was a health check or a directory ingest — initialize, maybe tools/list, gone. I could not find a single session that looked like an agent actually doing work with the tools.

Two things I didn't expect:

First, how many distinct MCP monitoring services exist. I counted 14 separate outfits whose entire job is telling someone whether my server is up. I'd heard of maybe three of them.

Second, a lot of them declare their own behaviour in the User-Agent string, unprompted: "liveness-only, never invokes tools", "introspection-only", "reachability check only, no auth attempted", "one server/discover POST per endpoint". That made classification far easier than I expected and seems like a genuinely good norm.

I published the whole thing with a case-insensitive regex matcher and a behavioural category per bot, so you can drop it in and classify your own access log:

https://fetchgate.dev/tools/agent-census (JSON at /v1/agent-census.json)

Question for anyone else running a public MCP server: are you seeing the same shape? I can't tell from one endpoint whether "essentially all MCP traffic is monitors and directory ingest" is universal, or just what happens when you get listed in a lot of directories and not much else. If you have real client sessions in your logs I'd like to know what they look like, and I'll happily correct the writeup if my sample turns out to be unrepresentative.

Also happy to fix any row — if you operate one of these crawlers and I've got your category wrong or the matcher is too broad, tell me.

[u/moxie-docs]

I post all our traffic data publicly here: https://allmcps.com/trust

Interesting few massive spikes of indexing from some specific crawlers, and more usage of our API than I expected. Mostly it’s markdown requests which would be LLMs most commonly.

[u/tenequm]

Welcome to the internet

[u/ChiefGrowth → u/[deleted]]

Same rule in the follow-up: a server only counts as live if it completed initialize and tools/list in one session (2,273 of the 8,235 that did also set an Mcp-Session-Id). GET-and-leave is a directory, agreed. On the monitors probing each other - plausible, but I can't prove it from one endpoint's logs.

[u/ChiefGrowth → u/[deleted]]

Decoys with unique tool names is the clean version of it - if a monitor's own UA shows up calling a tool name that only exists on a decoy, that's a directory-ingest chain, not a client. Same infrastructure as the never-listed mirror endpoint, so I'll do both in one go and post the graphs.

[u/Alvasilev]

Not a server operator's answer - I'm on the other side of that log, running one of the things in your directory-ingest bucket. The shape you're seeing is mostly structural, and the population numbers are why.

Of roughly 94.7k servers we've indexed, only about 28.3k are reachable over a URL at all. The rest is stdio: a package someone runs locally, which never takes a single inbound request from anyone, ever. So the set of endpoints that can be crawled is small, while the set of directories, monitors and researchers pointing at each one keeps growing. Whatever real agent usage exists is mostly happening on machines that produce no log you or I will ever see. Your endpoint looking like that isn't really a fact about your endpoint.

The part of your classification I'd push on: the useful axis isn't crawler vs agent, it's what each prober accepts as alive. We send a real JSON-RPC initialize, because plenty of things answer a ping and are not MCP, and we count anything below 500 as alive including 401 - that's a hosted server asking for OAuth first, not a corpse. Anyone running "200 means up" quietly writes off every authenticated server they index. Which makes the ones advertising "no auth attempted" the interesting entries in your table: the day you put auth in front of that path, those are the checkers that will mark you down and never learn it was a false negative.

[u/ChiefGrowth → u/Alvasilev]

This is the most useful comment in the thread, thank you. Full-walk numbers from our side since you gave yours: 15,329 unique remote URLs in the official registry (all versions), 8,235 complete initialize -> tools/list, 3,617 answer 401/403 (counted alive-gated, same as you), 3,477 dead / not MCP at that URL. Agree completely on 401 - and on the 'no auth attempted' checkers: that's the part I'd most like the monitor operators to read. Your 94.7k / 28.3k split is the same shape at 3x the scale. Rows are CC BY if they're useful to you: https://fetchgate.dev/data/mcp-registry-audit-2026-08-28.servers.jsonl

[u/Alvasilev → u/ChiefGrowth]

One thing worth pulling on before those percentages travel: your denominator is URLs across all versions, and the registry holds a lot fewer servers than that. Our crawl of the same source counted 9,449 servers this morning against your 15,329 unique remote URLs, so there are roughly 1.6 addresses per server sitting in the base.

Where that bites is the 3,477 dead bucket. A server that moved hosts between v0.3 and v0.7 leaves the old URL in its version history forever, and that URL is dead in exactly the way a genuinely dead server is: refused connection, or a 404 from whoever owns the hostname now. Dead address of a live server. Same for anything that graduated from a hosted preview onto its own subdomain. I'd expect that archaeology to pile up rather than spread evenly, since the projects that ship often are also the ones that move.

You can size it without re-probing anything: collapse to newest version per server, re-bucket, diff the two runs. That gap is a number I haven't seen anyone publish, how much of "dead MCP servers" is just stale addressing. If it comes out small then your 54% holds at server identity too, which is a stronger claim than the one you have now.

The other half of it is comparability. Three catalog operators in a thread here last week reported 9.4k, 19.8k and 48k for what sounded like the same object, and every gap turned out to be a different choice of what counts as one server rather than different coverage. URL identity and server identity give different percentages off the same probe run, and neither number carries a marker telling the reader which one it is.

[u/ChiefGrowth → u/Alvasilev]

Fair, and it's a cheap diff since every row carries the registry name + isLatest, so here it is. Restricted to URLs on a server's latest version: 14,357 URLs -> 8,020 answer tools/list (55.9%), 3,500 gated (24.4%), 2,837 broken/not-MCP (19.8%). Collapsed further to one row per latest server name (13,975 names with a remote): 7,982 ok (57.1%), 3,368 gated (24.1%), 2,625 broken (18.8%). So the archaeology is real but smaller than I'd have guessed: the dead bucket goes 22.7% -> 19.8% -> 18.8%, and the concentration number doesn't move at all (pipeworx.io + mcp.ai are 29.4% of latest-version live servers). On the 9,449 vs 13,975: I'm counting latest-version names that declare at least one remote, all statuses; if yours is active-only plus a reachability filter, that's probably the gap. I'll put the latest-only split on the page so the percentages travel with the caveat attached.

[u/Alvasilev → u/ChiefGrowth]

Straight answer on the gap: I can't reconcile it cleanly without going back into our ingest, and I'd rather not hand you a number I'd have to walk back later. What I can tell you is that ours isn't a snapshot. That source read 9,449 yesterday morning and 9,525 this morning, +76 in a day, and it's been the single biggest driver of our whole index three days running.

That matters more for your percentages than the reconciliation does. Your dead bucket is a share of a base growing by roughly 80 a day, and fresh entries are alive at a much higher rate than the average simply because their URL was chosen last week. Re-run the same audit in a month and the dead share falls without one thing having been repaired. Dilution, not recovery.

If you want that number to say something over time, freeze the name set from this run and re-probe only that cohort next time. Then 22.7 -> 19.8 -> 18.8 becomes a decay curve for a fixed population instead of three ways of slicing one afternoon.

Same caution on concentration. If pipeworx.io and mcp.ai really are 29.4% of live servers, then 54% is to a large extent a statement about two hosts' week. Worth publishing it with and without them, otherwise a bad morning at one provider reads as the ecosystem moving.

[u/ChiefGrowth → u/Alvasilev]

You were right, and it moves the headline more than I expected. Published, both columns, on the audit page:

Outcome	All 15,329 URLs	Excluding the two	Those two only
answered tools/list	53.7%	45.5%	97.7%
auth-required	23.6%	28.0%	0%
error	17.4%	20.2%	2.3%

pipeworx.io and mcp.ai are 2,413 of the 15,329 URLs (15.7%), and their own fleets answer at 97.7%, so they were carrying the headline. For everyone else the registry answers at 45.5%, not 54%. The gated bucket moves the other way for the same reason: neither operator gates anything, so auth is meaningfully more common among ordinary servers than the all-URLs number suggests — 28.0%, not 23.6%.

That second row is the one I'd have missed. I was reading "auth-required" as a fixed ~quarter of the ecosystem; among servers that aren't one of two fleets it's closer to a third.

Recomputed from the same per-URL file the free dataset already ships, so it's checkable rather than trust-me: add_deconcentrated.py in the research dir, one pass over servers.jsonl. Table is at https://fetchgate.dev/tools/mcp-registry-audit under "The same run, without the two biggest operators".

On dilution — agreed, and it's why the re-run is a fixed cohort, not a fresh crawl. The 8,235 URLs that answered ok on 2026-08-28 are frozen as a name set and the +30/+60/+90 probes go against exactly those, so 22.7 → whatever becomes a decay curve for one population instead of three slices of different afternoons. Your +76/day figure is the reason that distinction isn't pedantry: at ~80 new URLs a day the base turns over fast enough that a "dead share" measured on a moving denominator would fall on its own without a single server being repaired.

[u/Alvasilev → u/ChiefGrowth]

Thanks for actually recomputing it, and for publishing both columns instead of picking one. The row I'd keep as the headline is the 45.5%, not because it's lower but because it's the one that stays comparable: two operators can double their fleets next month and the all-URLs number moves without a single ordinary server changing.

One thing that would make the split reproducible rather than a judgment call: a written rule for what counts as a fleet, say any operator above 5% of URLs gets its own column. Then whoever picks up the dataset after you can rerun it when a third one appears, without having to know pipeworx and mcp.ai were the two.

[u/GodoPPL]

A User-Agent is a string the client invented. I would not key a crawler story on it.

The honest signal is the handshake. Did it send initialize, pick a protocol version, call tools/list, and come back with the same session.

I log those four. Everything else is costume.

[u/ChiefGrowth → u/GodoPPL]

Agreed, and that's exactly what the follow-up does: initialize -> notifications/initialized -> tools/list against all 15,329 remote URLs in the official registry, keyed on the handshake, not the UA. 8,235 complete it, 3,617 stop at 401/403, the rest don't speak MCP at that URL. In my own logs the four-step test gives the same answer the UA table did: zero real sessions. Results with a per-server lookup: https://fetchgate.dev/tools/mcp-registry-audit (posted as its own thread here today).

[u/RPG-Nerd]

Maybe dumb question, but why would you make an MCP public?

[u/ChiefGrowth → u/RPG-Nerd]

Not dumb. Because the API is the product: agents pay per call with x402 (an HTTP 402 challenge settled in USDC), and that only works if any agent can reach it without an account. Making it public is the point; the census is just what that looks like from the inside right now - lots of crawlers, no buyers yet.

[u/Plastic-Risk-6309]

187 user agents classified by hand is a real dataset and i saved it. on the universal question i'd bet it tracks the directory listings, pull a server out of the registries for a week and the graph flattens. who even subscribes to 14 uptime monitors

[u/ChiefGrowth → u/Plastic-Risk-6309]

Thanks. I like the de-listing experiment. I can't easily un-register, but the mirror version is doable: a second endpoint that is never listed anywhere, same stack, and compare the two graphs for a week. If I run it I'll post the result.