I published a survey of 4,749 public MCP server schemas a few weeks ago. The headline was that 17.7% of tools carry a description containing no word that distinguishes them from a sibling tool on the same server.
Then people who actually run MCP servers showed up in the comments, and over about ten days they took the measurement apart.
One found a bug. Several found a design flaw. Two of them found opposite design flaws that cancel each other out, which was the most useful thing that happened. And one found a ceiling the whole approach can't get past.
Here's all of it, because the corrections are worth more than the original number.
The bug: no stemming
The metric counts content words in a tool's description that appear in no other tool's description on the same server. It does exact string matching.
Someone pointed at two descriptions from his own server:
analyze_job_description "Extract what a job posting actually screens on."
optimize_resume "Rewrite a resume so it passes ATS screening."
screens and screening. Same word, two forms, and my counter treats them as unrelated — so these two tools score as distinctive on the one word that actually links them.
That's not a nuance. It's wrong, and it's wrong across all 82,549 tools. Stemming before counting is a small change with an unknown effect on the headline figure, which I'll report when I've re-run it.
The design flaw: lexical distinctness isn't ambiguity
The same author gave me his full ten-tool list and said three of them were confusable in practice. So I ran the metric on it.
50.0% score_resume "Score a resume for ATS compatibility."
60.0% analyze_job_description "Extract what a job posting actually screens on."
60.0% optimize_resume "Rewrite a resume so it passes ATS screening."
75.0% search_jobs "Return the user's job matches."
100.0% update_job_preferences "Set the roles and locations it hunts for."
His ambiguous cluster — score, analyze, optimize — comes out at 50–60%. My corpus calls that healthy.
My first fix was to compute distinctiveness on verbs alone, on the theory that verbs carry the action and nouns are shared boilerplate.
That fails harder. His ten verbs are score, extract, rewrite, produce, write, translate, return, set and build. All distinct. Verb-only distinctiveness scores every tool at 100%, including all three ambiguous ones.
The problem isn't granularity. Lexical distinctness and semantic distinctness are different properties, and no word-counting metric at any resolution closes the gap.
What does show the cluster is the shared input nouns — resume in four descriptions, job in three, ats in two. Those three tools are ambiguous because they act on the same object, not because they're worded alike.
His rule, better than my number: if two tools can ever be correct for the same sentence, you don't have two tools. You have one tool with a parameter.
So the fix isn't a better distinctive share. It's a second signal, measuring something else entirely. Distinctive share asks whether two descriptions look alike. Input overlap asks whether two tools could both be right.
The correction to the correction: overlap alone flags good design
Someone running a six-tool ticket server pushed back, with a pair he'd deliberately kept:
Batch retrieval and an analysis bundle. Identical input shape. Different intended output — flat details for more tickets, versus bounded evidence with optional comments and source anchors.
Input overlap would flag that pair. He'd be right to dismiss it, because the boundary is stated: a model reading those two descriptions has something to discriminate on even though the inputs are the same.
So overlap needs a partner. Overlapping inputs with distinguishable outputs is fine design. Overlapping inputs with indistinguishable outputs is the failure. Flagging the first is exactly the false positive that gets a signal switched off — and on a large server, a 10% false positive rate buries everything.
His test is better than any metric I can compute: can a model infer the intended output and boundary from the name, description and parameter contract alone, without knowing the repo or the author's intent?
The two corrections that fight each other
This is the pair I'd have got wrong silently.
One reader pointed out that the metric penalises focused servers. A memory server says "memory" in every description by design. A ticket server says "ticket". Those words count as shared, but they aren't a collision — they're the name of the server, repeated. Drop words appearing in more than 80% of a server's tools before counting, and small focused servers score higher.
Correct, and I was about to implement it globally.
Then the ten-tool author showed why that breaks the other signal. On a focused server, the collision is the domain noun. Strip resume from his descriptions and you delete the exact words that make score, optimize and analyze confusable. I'd have shipped a sharper distinctive-share number that quietly destroyed input overlap.
The resolution is two readings with different preprocessing, not one corrected score:
- Domain-stripped distinctiveness — do these descriptions look alike?
- Domain-intact input overlap — could two tools be right for the same sentence?
And the gap between them is the finding. A server that scores high on the first and badly on the second is focused, acts on one object throughout, and never says what its tools return. That's a common shape and it deserves its own line rather than being averaged away.
A third reader then improved the threshold out of existence. Frequency isn't a dial to tune — it's a distinction the same count already makes. A noun in nearly every tool names the server. A noun in a subset of tools names a cluster, and that subset is your candidate collision list.
Run it on the ten-tool server and the tightest group is ats, shared by score and optimize and nothing else. Which is the pair he flagged first. Same counter, no threshold, no tuning.
The ceiling
The same reader then found the thing none of this reaches.
analyze_job_description and score_resume share no word at all, stemmed or not. And they still collide, because "check my resume for this job" names both objects at once. The tools partition an intent that the sentence doesn't.
That collision exists between a request and a set of tools, not between two descriptions. Nothing computed on descriptions alone will ever see it.
That's a hard ceiling on static analysis, not a gap to close with a cleverer metric. The only artefact that catches it is a small labelled set of real requests — a handful of plausible sentences and which tool each should route to. Which is the one thing a corpus can't generate.
Two things I hadn't measured at all
Names. Two people independently said that when descriptions collide, models fall back to pattern-matching on tool names. One put it: "the model stops reading descriptions and pattern-matches on names, and then your error rate is really a naming problem."
That changes what a collision means. Gmail_DeleteDraftEmail and Gmail_SendDraftEmail have zero-distinctive descriptions — that pair was my headline example — but their names are clear. The names are carrying the load. A hypothetical pair with the same descriptions and names like delete_item and remove_item scores identically and is in far worse shape.
Annotations. MCP has readOnlyHint, destructiveHint, idempotentHint, openWorldHint. Two readers raised the same point: authors set them and write "Read-only" and "REPLACES" in the prose by hand, because the annotations are read by the client deciding whether to prompt for confirmation, while the description is read by the model deciding whether to call the thing. Two readers, two channels, and nothing requires them to agree.
Which makes a mechanical check available: descriptions mentioning constraints with no matching annotation set, and annotations set with nothing in the prose. The first means the client can't protect the user. The second means the model doesn't know a tool is destructive at the moment it's choosing.
What I'd take from this
Every correction came from someone who knew what the text was for.
I measured a property of the text and treated it as a proxy for ambiguity. Lexical distinctness looked like a proxy until someone showed me ten tools where it isn't. Populated descriptions looked like described parameters until someone pointed at one that says nothing. Descriptions looked like the whole interface until two people said names do the work when descriptions fail.
The original study was careful about one thing: it drew a hard line between what a schema shows and what models do with it, and refused to claim the second. That line held. What didn't hold was the assumption that measuring the text well is the same as measuring the thing that matters.
The published numbers stand as published — they measure what they say they measure, and they're all floors. The more useful metrics are the ones nobody had asked for yet.
Original study, data and analysis scripts: https://github.com/getmcpulse/mcp-schema-study
The schema checker runs these measurements on your own tools/list in the browser. The corrections above are being added to it.
I build MCPulse, an SDK that reports what models actually do with your tools under real traffic. If something here matches what you've seen on your own server, I'd like to hear it — every improvement above came from exactly that.
