I scanned 150,074 sites. The most common mistake is from 2005

Last time I wrote about building a scanner that refuses to score you for things you have not published. Plenty of people asked the obvious next question: fine, so what does the web actually look like when you measure it that way?

So I pointed it at the top of the Tranco list and let it run. Between 17 and 19 September it fetched the agent facing files and endpoints of 150,074 domains. Not a sample I picked. The list in rank order, until I ran out.

Here is what came back.

Rules for AI outnumber things for AI, eleven to one

Of the 150,074 hosts, 88,484 published something a machine can read. Everything below is counted over those 88,484 and never over the whole list, because a host that never answered is not evidence of anything.

• 34,160 have written a rule about AI crawlers, or published a signal about how their content may be used.
• 3,007 have published an interface an agent can call.

I want to be careful about that first number, because it is easy to read too much into it. It counts sites that named a crawler in robots.txt, or published a content signal, an ai.txt, or a mining reservation. It does not mean they blocked anything. A group headed by a crawler's name is just as valid with an Allow in it as with a Disallow, and from outside I cannot tell which.

So the honest version is: a third of the web that publishes anything has decided to have a policy about AI. Very few of them have built anything for it yet.

The thing I did not expect

I built this to measure new protocols. The most common fault it found is twenty one years old.

61,587 hosts publish an XML sitemap. 21,448 of them break one of the format's own rules, which is about 35%. The sitemap spec was agreed in 2005.

One fault accounts for roughly half of every problem the scanner found anywhere, across every technology it checks:

https://example.com/products/sitemap.xml

<url><loc>https://example.com/about</loc></url>
                              ^^^^^^
     the file sits in /products/, so this URL is outside its scope

14,176 sites do this. That is 23% of everyone with a sitemap.

sitemaps.org says a sitemap may only list URLs at or below its own folder. Google reads those URLs anyway if you submitted the file through Search Console, which is why so many people have never seen it break. Submit the same file any other way, or hand it to anything that is not Google, and the out of scope entries are dropped.

The rest:

Worth splitting by how much it costs you. 11,780 sites break only the folder rule, which Google forgives if you submitted it there. 9,668 are wrong no matter how you submit them. That is one in six sites with a sitemap.

If you take one thing from this post: go and look at your sitemap before you think about anything with "agent" in the name.

Which crawlers people actually name

33,626 sites name at least one crawler in robots.txt. The ones they pick:

CCBot at 24% is the one that gives the game away. Common Crawl has been around for over a decade and has nothing to do with the current wave. Seeing it that high, in that company, tells you a lot of these rules were pasted from a list someone found rather than decided by the site.

What the three per cent built

The 3,007 sites with something callable, by what they published:

Eight NLWeb endpoints, across 88,484 hosts that publish anything at all.

When these do break, they break in boring ways rather than dramatic ones. 176 of the 497 OpenAPI descriptions describe a value without saying what type it is. 112 leave an operation with no operationId, so an agent picking a call has nothing stable to name it by. Of the 73 A2A agent cards, 54 are missing fields the spec requires.

llms.txt is its own story. 13,730 sites publish one, which makes it the second most adopted thing in the whole study after the sitemap. 1,022 of those start with text before the title, which the format does not allow.

What this does not say

It is not an adoption rate for the web, and I would rather say so myself than have it quoted as one. The list behind it ranks domain names by popularity, not websites. 27,992 of these names never replied, another 14,227 turned me away, and a good chunk of the rest are not sites at all: nameservers, CDN endpoints, API hosts, parked domains.

Absence is still never counted as a fault, which is the whole argument of the last post. No MCP server card means nothing is reported. An MCP server card whose endpoint does not answer is a defect. Those two stay apart everywhere in this data.

And one caveat I would rather volunteer than be caught on. Each host was put into exactly one industry by a language model reading its home page. It placed 84,149 of the 88,484. The categories are mine, not a standard taxonomy, and I have not run an accuracy audit on them. So treat anything I say about industries as indicative, not measured.

Run it on your own site

Same checks, same rules, one command:

npx agentic-readiness example.com

It exits 0 when you have not adopted something and non zero only when something you did publish is broken. That is deliberate, and it means you can put it in CI without it failing your build over a protocol you have never heard of.

Agentic readiness of example.com

  OK    Access policies (robots.txt / ai.txt / TDMRep)
  OK    Sitemap

  2 of 14 technologies published, 0 with defects.

  Nothing published here is broken. Absence is never reported.

The full study has the per industry breakdown and every fault counted. The findings are published under CC BY 4.0, so quote them, chart them, build on them, including commercially, as long as you credit us.

Same offer as last time. Take it, argue with it, or tell me where I have got it wrong.