Your verification tool reports what it saw, never where it didn't look

Originally published on hexisteme notes.

I run a pipeline that turns short scripts into map animations and publishes them. The tail end of it is a verification stage: render the video, extract frames on a fixed interval, tile them into a contact sheet, eyeball the sheet, then fill in a compliance checklist — five automated checks and seven manual ones — and publish if everything passes.

Five episodes went through it. All twelve items passed on all five. They shipped publicly.

I audited those same five later and found four defects. One of them broke the central argument of an episode: the video's opening claim is that one country is larger than three others combined, and the overlay meant to demonstrate that stacked all three onto the same center point, so they covered each other instead of tiling into a combined area. The comparison the episode is built on never appeared on screen. Another put a number on screen that had no source anywhere in the project files.

The checks hadn't lied. Every one of them reported accurately on what it looked at. The problem is that a passing report is silent about everything outside its aperture, and silence reads as a pass.

I ended up with three distinct failures stacked on top of each other. They look identical in the report — a green check — and they need three completely different fixes. Confusing them is how I spent an afternoon proposing a change that would have made things worse.

Layer one: the places it never looked

My first reaction was that the sampling was too crude. Fixed-interval frame extraction is dumb — it grabs whatever happens to be on screen every N frames. The renderer emits an event log (marker_in, fill_complete, transition), so why not sample at the events instead? You can compute the exact frame an effect appears, and for anything animated, the exact frame of its cycle peak. Strictly more precise.

It also worked. I used that method to pin down one of the four defects to a specific frame number.

So I proposed making it the protocol. A colleague pushed back with one question: the other defect you found today — did you find that one from an event, or did you just look at a frame? Does that effect even emit an event?

I went and counted.

Six of nine types emit nothing. By instance count, 15 of 33 — 45% of everything on screen is structurally invisible to an event-driven scan.

And the part that settled it: both defects found that day were in the 45%. The effect type with the best event coverage, marker, was clean across all five published episodes. Every defect lived in the gap; nothing lived in the covered region.

So my "improvement" would have missed both of the defects that motivated it. Precision up, coverage down, net negative. The proposal was quantitatively refuted about forty-five minutes after I made it.

The lesson isn't that event-driven sampling is bad. It's precise, and I still use it — as a second pass alongside the interval sweep, not instead of it. The lesson is that the tool doesn't tell you what it's pointed at. You run the scan, you get results, and the results do not carry a line saying "18 of 33 instances were candidates; the other 15 were never eligible."

The fix for this layer isn't a better scan. It's making the coverage table above an artifact — something the scan emits every run, next to its findings. Without that table, "verified every effect via the event log" silently means "verified 55% of them," and there is no place in the output where the difference would show up.

Layer two: the places it looked, in a medium that couldn't answer

Four more mistakes came out of the same audit, all the same shape, and none of them are fixable by scanning more.

A ruler graphic had 100,915 km attached to it. The segment it was drawn across was about 1,010 km of real distance. A hundredfold mismatch. I wrote it up as the screen stating a falsehood and recommended a validation check comparing label magnitude against segment length.

Wrong. I hadn't read the spec to the end. The field wasn't label — a static caption — it was label_count_from: 28953 → label_count_to: 100915. Not a label on a segment at all: a count-up, synchronized with a camera zoom, so the number climbs while the view pushes in. The episode is about the coastline paradox. The graphic was performing the thesis — measure at a finer scale, get a bigger number — and the design intent was written down in the editor's note. I'd diagnosed a deliberate device as a bug because I recognized the shape of the widget and stopped reading.

A stamp_in event fired twice, once in the hook and once at the payoff. A stamp is a signature element in this format, so bookending felt like it could be intentional. I passed it as not-a-defect.

Wrong again, in the opposite direction. When I actually opened the code, the signature primitive (insignia) was pinned to the lowest tier and explicitly barred from promotion — it can't be what fires stamp_in. A different primitive fires it, sitting at the top tier, and that tier's definition read: T3 (aftermath) = the information an event leaves behind. The aftermath was playing in the hook, before the event that produces it happened. It was a defect, and I'd cleared it by reasoning from the event's name.

My colleague made the same class of mistake twice, in the other direction. Once, they inferred the schema's indexing convention — zero-based or one-based — by looking at which scene appeared in a contact sheet and working backwards. Once, they read a raw geographic data file and asserted what the renderer would draw, not knowing a filter sat between the file and the canvas.

Four instances, one sentence: an observation from one medium got promoted into a fact about another.

• name → definition (it's called stamp_in, so it's the stamp; the field is a label, so it labels)
• frame → schema (the screen shows this, so the convention must be that)
• source data → runtime behavior (the file says this, so the render will too)

Both directions matter. I did name→definition twice; they did frame→schema and data→behavior. It isn't one person's bad habit, it's a property of working across representations.

This layer looks like layer one — something got missed — but no amount of additional scanning fixes it. The frames were all there and I looked at them. A frame tells you what got drawn; it cannot tell you how a schema field is interpreted. The fix is a change of medium, not a change of coverage: to ask what something means, read where meaning lives. Which for a field name is the code that consumes it, and for a data file is the transform chain between the file and the output.

Layer three: the place it could see and was structurally steered away from

A marker's text card had its position bound to the radius of an animated pulse ring, so the card drifted outward on a 0.8-second cycle. At the top of the cycle it clipped off the frame edge.

This happened on every cycle, in every scene with a marker, in multiple published videos. It never appeared on a single contact sheet.

Markers appear 0.6 seconds after their scene starts. The pulse period is 0.8 seconds. Frames were extracted at a fixed interval that happened to be commensurate with the scene length, so the sampler landed on nearly the same phase of the cycle every time. Textbook aliasing — the same reason a wheel spins backwards on video.

Nothing was hidden. Every frame containing the defect got rendered, and the defect recurred more than once a second. The sampler just walked past the phase where it was visible, reliably, for weeks.

The fix here is neither coverage nor medium. It's sampling design: for anything periodic, compute the phase you care about and target it explicitly. Cycle peak, cycle trough, first frame after appearance. Uniform sampling of a periodic signal is a lottery you can lose the same way every time.

Three symptoms, three different repairs

They present identically — a check that passed — so it's easy to reach for the wrong one. I did exactly that: I found a layer-three problem and proposed a fix that would have deepened layer one.

How a cross-check quietly becomes a copy

Two of us were working this, deliberately checking each other. That's supposed to be the safety net, and here's where it tore.

When I passed along the ruler misdiagnosis, my colleague didn't just accept my conclusion — they started building a schema change on top of it, designing a validator for label-versus-segment mismatches. They weren't being careless. They were treating my premise ("the field is a static label") as settled and reasoning forward from it. Their work was an independent check on my conclusion and a faithful copy of my error.

Two legs that share a premise aren't verification. They're duplication with extra steps.

The transmission vector was specific, and it was summarization. I read the spec, compressed it to "the label says 100,915 km," and handed over the compression. On the receiving end, a summary doesn't arrive marked as a claim to be checked — it arrives as ground to stand on.

The rule I took from it: when a field is load-bearing for your argument, paste its definition, don't describe it. This actually worked later the same day. Instead of summarizing the tier system, I pasted the tier table verbatim — the literal dict with its comments. My colleague confirmed the aftermath-before-event defect without reopening the file, and their confirmation was genuinely independent of my reasoning, because what I'd sent them was evidence rather than a conclusion.

Then, while writing this piece, I did it to myself with no colleague involved.

One of the unsourced numbers was a national coastline length, and the open question was whether it included a distant Arctic archipelago — which matters, because the map renders that archipelago in the same highlight color as the mainland. The authoritative sources were unreachable from my network, so I pulled a reference page through a fetch tool that returns a summary of the page rather than the page. The summary said, cleanly, that the measurement was specified as excluding the archipelago.

That would have closed the question. Instead I pulled the raw text, because the answer felt too tidy for something I'd already failed to resolve twice. In the raw text, the exclusion note is real — attached to a land-use-by-area table from a different section. It has nothing to do with the coastline figure. The summarizer had joined two true statements from two parts of the document into one false one.

The conclusion I'd have drawn from the summary is still probably correct; a different sentence in the raw text supports it by a different route. That's the trap closing exactly as described above: the wrong method would have produced the right answer, and the rightness would have been mistaken for verification.

What's worth noting is that no human and no colleague was in this loop. Every search tool, retrieval layer, and documentation assistant in your stack returns summaries, and a summary does not arrive labeled as a claim to be checked — it arrives as a premise. The two-legged review that quietly becomes one leg doesn't need two people. It just needs one summarizer in the middle.

This shape is everywhere

Verification tools report what they examined and say nothing about what they never reached. A passing report renders that silence as a pass.

Once you're looking for it:

• Test coverage. 92% line coverage means 92% of the lines that got executed by tests you wrote. Behavior with no test at all isn't in the denominator — it isn't failing, it's absent. The number can go up while a whole subsystem stays untested.
• Monitoring dashboards. You see the metrics you added. Outages arrive through the ones you didn't. A green dashboard and a healthy system are separate events that usually coincide.
• Type checkers. A clean run means no type errors were found in the checked code. Going from "types check" to "the code is correct" is a medium change (layer two), and the Any at your I/O boundary is a coverage hole (layer one).
• Log sampling. Sample 1% of requests and if the sampling interval resonates with a cron schedule, an entire class of job can be invisible indefinitely (layer three).
• Code review. A diff tells you what changed. It doesn't tell you what the system does — which is why a correct-looking diff can be reviewed carefully and still break something nobody looked at.

Three questions I now ask of any check that comes back green:

1. What is this check's denominator? Not the pass rate — the candidate set. "45% were never eligible" appears nowhere in a 100% pass rate.
2. Does the medium I observed in actually contain the answer to my question? A name is not a definition, a screenshot is not a schema, a source file is not a runtime.
3. Is a summary I passed along becoming someone else's premise? If so, that review is one layer deep no matter how many people signed off.

Passing and being traceable are different events

One more, and it's the one I'd keep if I could only keep one.

The audit turned up three numbers displayed on screen with no supporting source in the project files. Two of them turned out to be correct. One wasn't traceable at all. All three got in through the same hole — the checks read the narration text, and no check read the numbers baked into graphics.

At the time I wrote down that two of the three were "harmless, just untracked." Then I went and actually sourced them. One is a fjord length, 205 km, which the reference gives — alongside a note that under a wider measurement scope the same fjord is 226 km. That episode is about how measurement scope changes length. The number it cites had the exact ambiguity the video was made to explain, and I'd have shipped it without the qualifier.

I had declared it harmless before looking. It wasn't a verdict; it was a guess wearing a verdict's clothes, and it happened to be a guess about the one thing the episode couldn't afford to be sloppy about.

A value passing your check and a value you can trace are different events. The two that were right weren't controlled outcomes, they were luck — and one of them, on inspection, wasn't even that.

More notes at hexisteme.github.io/notes.