Building an honest scanner for the agentic web

Why I started

Every few weeks somebody sends me an "AI readiness score" for a website. It is always red. It never explains what the missing items would actually do for that site, and it never distinguishes between two completely different situations:

1. You have not published an emerging protocol.
2. You have published one, and it does not work.

The first is usually an observation. The protocol may not apply to your site, its adoption may be negligible, real agents may not touch it, and waiting may be the correct decision. The second is a genuine defect: an agent card advertising an endpoint that answers nothing, a discovery document that is not discoverable, a required field that is missing.

Collapsing those two into one number is how you manufacture urgency. A low score creates panic without explaining whether anything is at stake. A high score hides real breakage, because unrelated features compensate for broken ones. Both are dishonest, and both are commercially convenient, which is presumably why they are everywhere.

There was a second motivation, closer to home. The people who want these technologies are usually SEOs and content teams. The people who can ship them are engineers with a roadmap full of platform work. So a finding sits in a backlog for two quarters, or an agency charges a lot of money for a file that takes ten minutes to write. Telling someone what is broken is only half a product. I wanted the other half.

So the goal became: measure honestly, then help implement.

What I built in the container

The core is a Lumar custom metric container (inputType: Puppeteer), organised into six categories:

Thirty technologies in total. 23 are scored. 7 are watched and deliberately not scored. That split is the product, not an implementation detail.

The contract: detect, then validate, then report

Every technology emits the same shape:

<tech>Present   boolean   the detection gate
<tech>Valid     boolean   present and defect-free
<tech><Defect>  boolean   one per known defect kind, true only when present-but-wrong

The rule that falls out of this is the one the whole container rests on: if a technology is not present, the collector computes nothing and emits nothing. Not Present: false. Nothing at all. There is no point validating, storing, or reporting anything about a technology a page never adopted.

That produces 309 per-URL metric fields and 185 site-wide fields, backed by 111 catalogued defects. Every defect names the clause and the revision it comes from, and whether the authority is a standard or a Lumar recommendation. We do not borrow a specification's authority for our own preferences, and the catalogue makes that distinction machine-readable rather than a matter of tone.

Two phases, because two kinds of fact

Page-level signals (canonical, JSON-LD, server-side rendering, Markdown twins, WebMCP) vary per URL and are read in the request handler. Site-wide signals (robots.txt, ai.txt, /.well-known/*, live protocol probes) are one fact per host, so they run once per host in a postCrawl handler and emit a crawl-level item. Nine site-wide signals, one row per host per signal.

Getting this wrong is expensive. Fetch a well-known file once per URL and a 50,000-page crawl knocks on somebody's server 50,000 times for a file that has not changed.

Naming evidence states, and never faking one

This is where most of my early false positives came from, so it is now a hard rule.

A fetch outcome is a discriminated union: complete, truncated, absent, unreachable. Only complete carries content, so a body we stopped reading at a byte cap cannot be handed to an analyzer, because the type does not expose it.

And more importantly:

Only an HTTP response proves absence. A 4xx or 5xx is the server saying "not here". Everything else, our own timeout, a spent budget, a reset connection, a blocked destination, is us failing to look, and must map to that signal's "unassessed" state.

Collapsing those convicts a site of our own timeout. That is the exact false positive this container exists to avoid, and the type system now refuses to let me write it.

Two registries that answer different questions

• src/defect-catalog/ says what can go wrong with a technology and on whose authority. One entry per emitted flag.
• src/tech/ says what the technology is: name, summary, adoption level, named adopters, the documents it is pinned to, the fixtures that demonstrate it.

Everything public is generated from that pair. The scanner, the documentation pages, the survey tables, and the report catalogue all come out of the same two files, so they cannot disagree with the code. When I add a defect, its documentation, its report column, and its public reference page appear because they are derived, not because I remembered to update three places.

Keeping up with specs that move weekly

This is the part I did not anticipate, and the part I am most pleased with.

These are emerging standards. MCP has shipped multiple revisions. Content Signals appeared and gained real adoption inside a year. Proposals turn up with a website, a GitHub repo and an enthusiastic announcement, and some of them die quietly.

A container pinned to a spec revision goes stale in weeks. Manual re-checking does not survive contact with a roadmap. So I built three Claude Code skills, versioned in the repo alongside the code they maintain. Each one has a bounded remit and writes to a bounded set of files, which is what stops an audit quietly becoming a refactor.

/check-correctness: is what we score still what the spec says?

Re-reads each technology's authoritative document and compares it against what the container actually validates. It hunts four kinds of drift:

• a bumped spec version
• a renamed or restructured field
• a new present-but-wrong surface we should be catching
• a check the spec no longer justifies

It is an audit, not a rewrite. It proposes changes and updates only the _Last verified_ stamps. Confirmed drift is implemented separately through /add-metric-checker, which does the full end-to-end job: interface field, collector, e2e fixtures, SOURCES.md citation, and the expectations gate.

Keeping "find the drift" and "fix the drift" as separate skills matters. An audit that can also refactor is an audit you cannot trust the output of.

/scan-emerging-tech: what should we admit next?

The intake step. It horizon-scans for technologies not yet in the container, and re-tests the "landscape only" and "dropped" rows for anything that has since matured.

Every candidate goes through an admission rubric:

• On-thesis. Is it part of a website's public agent-facing surface? Enterprise data-lake architecture is not.
• Crawl-detectable. Can we observe it from outside, without credentials?
• Present-but-wrong surface. Is there a way to adopt it and get it wrong? If not, there is nothing for us to score.
• Distinct. Does it overlap something we already check?
• Spec-anchored. Is there a document to cite, or only a blog post?

This rubric is why the container scores 23 things and watches 7 rather than scoring everything with a logo. It is also why some things were dropped: ACP and AP2 came out because neither presented a crawl-detectable present-but-wrong surface, and llms.txt came out for lack of adoption. Both decisions are recorded with reasoning, because "we deliberately do not score this" is a claim we should have to defend.

/track-adoption: who is actually using this?

Maintains an adoption ladder (L0 to L4) for every technology we score or watch, with trajectory, named adopters, depth of implementation, and dated citations. It writes to one document and nothing else.

Its evidence feeds the scan rubric's adoption gate, which closes the loop: a watched technology maturing shows up here first, then routes to the scan skill, then to the checker skill. And a scored technology whose adoption collapses triggers a drop discussion.

Today's ladder, straight from the registry, tells a useful story on its own:

L4  established   robots.txt, XML sitemap, canonical, JSON-LD, schema:Action,
                  Product schema, editorial metadata, OpenAPI, Link headers,
                  language and encoding, server-side rendering
L3  widespread    AI-bot rules, Content Signals, UCP profile
L2  adopted       MCP authorization, Markdown twin, TDMRep, Web Bot Auth
L1  emerging      MCP server card, A2A agent card, WebMCP, NLWeb
L0  negligible    ai.txt

The tall bars are two decades of web standards. The short ones are this year. A tool that treats those as equally urgent is not measuring readiness, it is selling anxiety.

Running it in the Lumar crawler

Inside a Lumar crawl, the container produces two kinds of output.

Per-URL metrics land on every page row, so you can segment and filter like any other crawl field. Site-wide items land in the crawl-level table, one row per host per signal, so a robots.txt fault appears once rather than on every one of a million URLs.

On top of that sit 33 predefined reports, and every one of them obeys the same rule:

A report's filter must AND-in <tech>Present eq true before any defect predicate.

A URL enters a report only when it actually adopted the technology. There is no report anywhere in the product that lists pages for not having something. Each report also shows every one of that technology's defect columns, ordered by severity, so no defect is invisible in the grid. That ordering is derived from the catalogue and asserted in tests, because I had eleven reports inherit the interface's declaration order, which is a parser's order, not a reader's.

Turning the checkers into a free scanner

The reports are for customers running crawls. But the argument I wanted to make is a public one, so I built a free scanner: enter a domain or a URL, get an honest answer, no account.

The important design decision is what it does not do. It does not reimplement the rules. It imports the container's own collectors and analyzers directly. The free tool and the paid audit run the same code, so they cannot drift apart and start disagreeing about what a valid MCP card looks like.

The output mirrors the container's shape: what you publish, whether each published thing is correct, and, when we could not test something properly, a plain statement that it was only partly assessed. That third state matters. A card declaring gRPC cannot be verified with an HTTPS request, so we say so instead of quietly certifying it.

There is no total score anywhere in the interface.

Builders: the other half of the product

Finding a fault and leaving you with it is the behaviour I was complaining about at the start. So every technology the scanner checks ends the same way: with something to do next.

Six builders generate a finished file: AI-crawler rules, Content Signals, ai.txt, TDMRep, an A2A agent card, an MCP server card. They interview you for what the site cannot tell us, read what you already publish so the form arrives half-complete, and generate the artifact.

Then the rule that makes it trustworthy:

Every generated artifact is run through the analyzer this container scores live sites with, and anything that fails is withheld with the reason.

We are not going to be the tool that emits the defects it reports. If a card would advertise an endpoint that does not answer, we do not write the address until the endpoint answers.

Seventeen more are tutorials, for technologies that cannot be handed over as a file because they need a running server or a template change: canonical tags, JSON-LD, Markdown twins, MCP authorization, NLWeb, WebMCP, server-side rendering. These are assembled from artifacts the repository already tests, the correct and broken fixtures from the e2e suite, and the defect catalogue's clauses. A tutorial cannot drift from what the scanner scores, because it is made of the same material.

Seven technologies we refuse to build, listed by name with the reason. "We track this proposal but do not currently recommend implementing it" is a more useful answer than a missing menu item.

One thing I got badly wrong and had to redo properly: the robots.txt builders originally emitted a fragment, while every publishing recipe said "serve this artifact at /robots.txt". Following the Next.js instructions would have replaced a site's entire robots.txt with a single line, deleting the wildcard group, every rule, every named crawler and the sitemap. A fragment also cannot express "change this crawler from Allow to Disallow", so asking to block a crawler the file already allowed returned "nothing to add".

The fix was a line-level editor that produces the complete file: your current robots.txt with the change applied, every comment and ordering choice preserved, and a list of exactly what changed. It splits a crawler out of a shared group rather than changing its neighbours, and it validates the file before and after so a fault you already had does not get blamed on us.

Eating the same food

It would be strange to publish a scanner that grades agent-readiness from a website only humans can use. So the scanner publishes everything it tells you to publish, and its own pages are scored by the container in CI.

• An MCP endpoint with three tools: scan a site, search the documentation, build a file. No account, no key.
• An A2A agent card and an OpenAPI contract.
• WebMCP registrations in the page, so a browsing agent can complete a builder wizard while you watch.
• A Markdown twin of every page, served by content negotiation and at its own address.
• Our own robots.txt with named AI-crawler groups, Content Signals, ai.txt, and a TDMRep reservation that matches what the Content Signals line says rather than contradicting it.

The three MCP tools compose into the actual job, which is the point of exposing them: find what is broken, read why it matters, generate the fix, check the fix against the same rules the scan uses. An agent can do the whole loop without anyone opening the page.

This is not decoration. It caught a real defect in code I had written an hour earlier. A new page declared itself a TechArticle with no author and no datePublished. TechArticle is in the Article family, so the container's own editorial checks fired: editorialMissingAuthor and editorialMissingDatePublished, two of the defects we report on other people's sites, published on our own page.

The structural fix was a test that scores every hand-written page through the container and asserts that no defect fires, so the next page cannot ship unscored. That is what dogfooding is for. Not the marketing line, the failing test.

Where it stands

The container ships 23 scored technologies and 111 catalogued defects, with 1,426 tests. The scanner adds 607 unit tests and a browser suite. Three maintenance skills keep the specs, the intake and the adoption evidence current, on a cadence rather than on a good intention.

The position underneath all of it is simple enough to fit in a sentence: missing is not the same as broken, and a tool that cannot tell the difference is not measuring your website, it is measuring your anxiety.

If any of this is useful to you, take it, argue with it, or tell me where I have got it wrong. The reasoning is all public, including the parts where I changed my mind.

Links

• The argument: https://agentic-readiness.lumar.io/mission
• The scanner: https://agentic-readiness.lumar.io/
• The reference, with every check, its authority, and a correct and broken example of each: https://agentic-readiness.lumar.io/docs