Your validator must measure what you ship, not what you loaded

Originally published on hexisteme notes.

I built a gate to catch a specific kind of lie. One narrative device in a pipeline that renders short map-explainer videos places country B next to country A "at the same scale," so a viewer can compare their sizes at a glance. Get the scaling wrong and the device becomes the lie it exists to prevent. A predecessor note covered a caption-width guard that passed every test because it measured text the renderer never actually drew. This is the same mistake in different clothes — a validator checking the wrong artifact — except here the artifact is geometry, and the gate didn't just miss a lie. It accused an already-published, honest video of telling one.

The device, and why it needs a guard

The renderer draws on an equirectangular (plate carrée) projection, where a degree of longitude compresses by cos(latitude) — move a country to a different latitude and its apparent width changes even though the country didn't. The renderer already corrects for this: lon_scale = cos(source_lat) / cos(dest_lat) inside translate_polygons_to_centroid in src/shorts_factory/map_scenes.py redraws the moved country as if it had always lived at the destination latitude, so it takes the same 1/cos φ inflation as the target and the ratio between them stays honest.

But that's one scale factor from one centroid latitude, applied to a whole polygon. It cannot unshear a country that spans 40° of latitude — the top and bottom inflate by different amounts, and one factor can't know that.

So I added a gate at script-ingest time, before any rendering happens:

residual = | on-screen area ratio / true area ratio − 1 |
reject if residual > 0.05

On-screen area is the planar shoelace area of the laid-out polygons; true area is a spherical integral using Earth's mean radius, 6371.0088 km — pure coordinate math, no frame required. The threshold is OVERLAY_AREA_RATIO_RESIDUAL_MAX = 0.05 in src/shorts_factory/script_schema.py.

To specify it, I hand-computed four golden vectors and told the implementing worker to reproduce them. It matched all four to four decimal places. I was confident. I shouldn't have been.

Running it against something already on the internet

I didn't stop at fixtures. I ran the finished gate against an episode that had already shipped: scene 0 of mongolia-empty places France next to Mongolia (MNG←FRA) for a size comparison. The gate's verdict and what a viewer actually sees disagreed by an order of magnitude:

That video is public. That scene is honest. The gate was wrong, and wrong in the least comfortable direction a validator can be: confidently, specifically, about something real that had already shipped.

Two ways to load the same file

The codebase has two ways to turn the same GeoJSON file into polygons: a raw parse that reads the file exactly as it ships, and load_world_geojson(), which parses and then calls _filter_far_flung_polygons() to drop pieces sitting far from a country's mainland — built for camera framing, so a bounding box doesn't get dragged across an ocean by a speck of overseas territory. The renderer calls load_world_geojson().

The gate called the raw parse. My reasoning felt airtight: to measure a country's true area you need its whole geometry, not a filtered subset. That's exactly backwards for the two countries where it mattered most.

Russia's raw polygon set crosses the antimeridian — Chukotka sits on the far side of the date line, so the coordinates span the full −180° to 180° range. Once that happens, both the shoelace-area formula and the centroid stop meaning anything; the shape isn't the shape anyone intends.

France fails more quietly. French Guiana, in South America at about 4°N, drags the combined centroid to (−10.7°E, 35.4°N) — the middle of the Atlantic, nowhere near France. Anchor the translation math to that point and the scaling is wrong before the comparison starts.

The renderer never draws either piece. The mongolia-empty script had include_overseas_territories: false, so the France on screen was metropolitan France, full stop. I had measured the honesty of a picture using geometry the picture never contained.

Why four golden vectors didn't save me

I'd computed all four golden vectors by hand, from the same raw geometry the gate used. The worker implementing the gate reproduced them to four decimal places — precision that's exactly what makes the failure worth writing down. Reproducing my numbers meant reproducing my misunderstanding. A golden vector proves an implementation matches its spec; it cannot prove the spec is correct. When the vectors and the spec share an author and an assumption, they don't cross-check each other — they agree with each other, and the agreement feels like confirmation.

What actually caught the bug wasn't a vector. It was pointing the finished gate at an artifact that already existed and was already known to be correct, and watching it fail anyway.

What the fix did to the conclusion, not just the code

The fix is one substitution: measure with load_world_geojson() plus that script's own include_overseas_territories flag — the exact geometry the renderer will draw for that episode, no more and no less.

I didn't stop at the one failing case. I re-derived the whole thing exhaustively (report/scripts/recount_honest_pairs_render_geometry.py) across all 177 countries and every pair whose true area ratio falls between 0.5 and 2.0 — 6,478 pairs, the band where a size comparison is actually a visual claim. Under the corrected geometry:

The worst pairs are all Antarctica — CHN←ATA 79%, BRA←ATA 76%, AUS←ATA 75% — and the rest of the 1.9% that fail concentrate in Antarctica and Russia, the countries with the most extreme latitude span. A few representative pairs show the shape of the correction:

Read the second and third columns together: Kazakhstan←Mongolia moves centroids a mere 0.1° apart and still lands at 2.5% residual, worse than India←Sweden's 38.8°-apart 0.8%. Centroid distance doesn't predict the residual — the cos correction already removes that axis. What's left is each country's own latitude span: Russia, running 41°N to 81°N, fails both pairs above regardless of how close the other centroid sits.

Before the fix, my own report on this device had concluded it was honest "only in very limited cases" — not a finding about the world, but an artifact of the instrument: a broken measurement doesn't only let bad things through, it also invents constraints you don't have. The same blind spot showed up again when a review proposed a simpler guard instead of the residual math — just keep the two centroid latitudes within ±5° of each other, a rule that sounds conservative because it's tight and cheap to explain. Measured against the same 6,478 pairs, the ±5° band passes only 828, against 6,353 (98.1%) for the residual gate, and it rejects 5,527 pairs that are honest by the actual area math while still letting 2 dishonest ones through. It isn't conservative on one side of a tradeoff; it's wrong in both directions at once, and the only way to see that was computing both rules over the same data instead of trusting which one sounded safer.

Pinning the regression where the invariant actually lives

The obvious regression test is "call load_world_geojson, not the raw parser." That pin breaks the moment either function gets renamed, and it doesn't say what actually needs to stay true. What has to hold is a relationship, not a function name: the gate's verdict has to track the render setting.

@pytest.mark.parametrize("include_overseas, should_raise", [(False, False), (True, True)])
def test_overlay_area_ratio_follows_include_overseas_territories(...):
    # same MNG←FRA pair: False -> residual 1.0%, passes / True -> Guiana returns, 14.4%, rejected

Same pair, same script, one boolean flipped, and the verdict flips with it — a smaller, more durable promise than "always call the right function," and the one the parametrized test in tests/test_script_schema.py actually pins down.

Four things worth carrying into your own validator

1. Check the validator reads what the artifact is actually built from. When a codebase keeps a "before cleaning" and an "after cleaning" version of the same data, a validator can quietly grab the wrong one — risk peaks when the two names look alike, like parse_x next to load_x.
2. Distrust the word "true." Raw input feeling truer than processed input is precisely the trap here. Nobody looking at the output is deceived by your source file — if anyone's deceived, it's by what renders.
3. Golden vectors validate conformance, not correctness. Vectors and spec sharing an author and an assumption aren't independent — they're the same claim written twice. Add a cheaper, separate check: run the new gate against artifacts already known to be good. Confirming everything that used to pass still passes is close to the cheapest test there is, and it's the one that actually caught this.
4. "Implemented to spec, golden vectors reproduced" is a claim about compliance, not correctness. Treat those as two separate claims a worker can hand you — only one tells you the spec itself was right.

Where this stops being true

This reasoning breaks if the geometry the renderer drops is routinely part of the claim being made. If narration says "all of Canada" while the screen shows the 84.2% that remains once the Arctic archipelago is filtered out, a screen-based gate happily passes that lie — it was never built to catch it, because for this device the screen was the right thing to measure. That failure mode doesn't need a stricter threshold; it needs a second, independent check asking whether the screen represents the claim, a different question from whether the screen is internally consistent. For an overlay size comparison, measuring the screen is the right call. The day narration starts making absolute-size claims instead of relative ones, this judgment needs revisiting, not reuse.

More notes at hexisteme.github.io/notes.