6 of 40 sites were pointing Google at a sitemap URL that returns HTML

A robots.txt Sitemap line that resolves to 200 is not the same thing as a readable sitemap. I found out the expensive way, on six sites at once, and the bug had been live for months.

The failure mode: 200 + text/html

$ curl -sI https://example.com/sitemap.xml.gz
HTTP/2 200
content-type: text/html; charset=utf-8

That is the entire bug, and it is completely invisible to normal monitoring.

The chain is boring and extremely common:

1. The site was originally built with a generator that gzipped its sitemap, so robots.txt got Sitemap: https://example.com/sitemap.xml.gz.
2. The build changed (Hugo out, a SPA build in). The .gz file stopped being produced.
3. The web server has the standard single-page-app fallback:

location / {
    try_files $uri $uri/ /index.html;
}

1. A request for a file that no longer exists therefore returns 200 OK with the app shell as HTML.

The crawler asks for a sitemap, gets a web page, and gives up. Silently. No 404 in the access log to alert on, no error in the build, and the submit button in Search Console still says "Success".

Six of the forty sites I look after had this exact robots.txt line. Every one of them returned a green 200.

Why status-code monitoring never catches it

• A synthetic check that asserts status == 200 passes.
• An uptime monitor passes.
• A "sitemap submitted successfully" message in Search Console passes — submission is just a queue write.
• Even a page-level error tracker passes, because nothing errored.

The only thing that noticed was Search Console URL Inspection, which reported entire sites as unknown to Google. That looks like an authority problem, so the first instinct is to buy links. That would have been money spent on the wrong problem: the discovery channel itself was broken.

Check 1 — parse the sitemap, don't fetch it

Never assert on the status code alone. Assert that the bytes parse as XML.

import re, urllib.request, xml.etree.ElementTree as ET

UA = {"User-Agent": "Mozilla/5.0 (compatible; sitemap-audit/1.0)"}

def sitemaps_from_robots(host):
    req = urllib.request.Request(f"https://{host}/robots.txt", headers=UA)
    txt = urllib.request.urlopen(req, timeout=20).read().decode("utf-8", "replace")
    return re.findall(r"(?im)^\s*sitemap:\s*(\S+)", txt)

def assert_real_sitemap(url):
    r = urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=30)
    body = r.read()
    ctype = r.headers.get("content-type", "")
    if "xml" not in ctype:
        raise AssertionError(f"{url}: wrong content-type {ctype!r} ({len(body)} bytes)")
    try:
        root = ET.fromstring(body)
    except ET.ParseError as e:
        raise AssertionError(f"{url}: not XML — {e}; first bytes {body[:80]!r}")
    return root

Two details matter here:

• content-type is a cheap early exit, but it is not sufficient. I have seen application/xml served for an error page.
• ET.fromstring is the real assertion. "The string starts with <" passes for <!DOCTYPE html>, which is precisely the case we are trying to catch.

Six sites, six failures, thirty seconds of runtime. That one function was worth more than every link-building experiment I ran that quarter.

Check 2 — a sitemap full of soft 404s

Fixing the .gz path made the sitemap readable, and immediately exposed a second layer: the SPA fallback means any URL returns 200. So a sitemap can list thousands of URLs that are not files, do not correspond to a route, and render the generic shell.

Mine had 385 entries. 188 of them were fake.

The tell is that a soft 404 renders identical content to the fallback. So compare a fingerprint, not a status:

import hashlib, re

def fingerprint(html):
    m = re.search(r"<title[^>]*>(.*?)</title>", html, re.S | re.I)
    return hashlib.sha1((m.group(1) if m else "").encode()).hexdigest()

shell = fingerprint(open("index.html").read())

for loc in sitemap_locs:
    body = urllib.request.urlopen(urllib.request.Request(loc, headers=UA)).read().decode("utf-8", "replace")
    if fingerprint(body) == shell:
        print("SOFT-404:", loc)

For a React/Vue SPA this catches the majority of cases for free. For a pre-rendered site it is stricter: each route should have its own <title>, so equal titles means the route was not actually pre-rendered.

Result after pruning: 385 → 197 entries, all of which resolve to a distinct page.

Check 3 — wherever canonical points, that URL must return 200

This one I consider the most dangerous class of bug, because it does not look like a bug at all.

A page declared <link rel="canonical" href="https://example.com/en/">. Requests to /en/ returned:

HTTP/2 403

The directory existed. It contained only subdirectories. There was no index.html, and autoindex was off, so nginx answered 403. The homepage was telling Google: the authoritative version of me is that other URL, and that URL refused to serve anything.

In effect the site had been pointing its entire weight signal at a dead address, and the language entry point was closed to users as well.

Static HTML checks will not find it, because the canonical tag is injected by JavaScript. You need a rendered DOM and then a second request:

const { chromium } = require("playwright-core");

for (const url of pages) {
  const page = await browser.newPage();
  await page.goto(url, { waitUntil: "networkidle" });
  const canonical = await page.evaluate(
    () => document.querySelector('link[rel="canonical"]')?.href || null
  );
  if (!canonical) { console.log("NO CANONICAL", url); continue; }

  const res = await page.request.get(canonical);
  if (res.status() !== 200) {
    console.log("DEAD CANONICAL", url, "->", canonical, res.status());
  }
}

That check takes about two seconds per page and is the single highest-value rule I added. It has since caught the same mistake on two other sites, both of which had a healthy-looking <link rel="canonical"> pointing at a 403.

Check 4 — probe language directories as their own URLs

Directory roots are not covered by sitemaps that only list article URLs. After finding one 403, I scanned 119 domains × 12 language prefixes (1,428 URLs) and found exactly one affected site — but that one was affected at both /en/ and /zh/, and it was the site whose root canonical pointed there.

If your site is example.com plus example.com/en/, treat those as four independent things to verify: status code, rendered <html lang>, <title>, and self-referencing canonical. A directory that renders correctly at / proves nothing about /en/.

How the two page types differ in practice

The audit covered both fully static pages and pre-rendered SPA routes, and the failure distribution was lopsided: static pages essentially never soft-404'd, because they are files. SPA routes failed whenever the pre-render step silently skipped a route and the fallback served the shell.

Two examples from that audit, if you want to compare the HTML:

• A statically pre-rendered article — one file, one title, no hydration step that can drop content.
• A route inside an SPA shell — pre-rendered HTML that then hydrates. This is the shape that produces soft 404s when the pre-render step misses a URL, and the shape where a static-only audit will happily report 200.

The lesson was not "SPAs are bad". It was that 200 means nothing in an app that returns index.html for everything. Every check has to assert on content, not on transport.

The pipeline I ended up with

Four layers, each one cheap enough to run on every deploy:

1. Static — every generated page: HTTP status, exactly one analytics tag, self-referencing canonical, title/H1 present, no placeholder strings.
2. Rendered — real headless Chrome: main content width, horizontal overflow, CTA visibility, and the canonical target returns 200.
3. Server-side — sitemap loc set vs. files on disk, lastmod not stale, no orphan pages, robots.txt Sitemap lines point at real XML.
4. Fleet — resolve robots.txt → sitemap for every domain, parse as XML, record the ones that need waiting on a CDN cache to expire.

The rule I keep coming back to: if a check only prints a value, it is not a check. It needs a threshold and a non-zero exit code, or it will pass forever while the site is invisible.

If you take one thing from this: go curl your robots.txt, pull the Sitemap line, and check the content-type. Six out of forty had it wrong, and none of them knew.