The Null Result Belonged To The Other Hypothesis

Originally published on hexisteme notes.

I spent half a day being wrong in writing, and the fix took three lines. The interesting part is not the fix. It is that I had already run the experiment that disproved my diagnosis, looked at its result, and recorded it as support.

The system is a renderer that animates a camera across a map: each scene declares a bounding box, and the camera eases from the previous scene's box to the next one. A quality check found stretches where the output frames contained no map at all — dark rectangles with only the caption text on them, totalling 0.93 seconds across three transitions in a 23-second clip.

I diagnosed it as unavoidable. I wrote that down, with a trade-off table and a falsification condition, in a document whose whole purpose is recording honest engineering judgment. Being wrong in that particular format is worse than being wrong casually, because the format makes it durable.

The hypothesis, and the experiment that "confirmed" it

My theory: there is nothing to draw. One of the scenes frames a 20-kilometre island near the end of a remote archipelago. To get there the camera passes through intermediate scales — a few hundred kilometres wide — and at those scales, in that part of the world, there is genuinely nothing but open water. Empty frames were therefore geometry, not a defect.

The test was obvious. The renderer takes an explicit list of what to draw per scene. I added two continents to the island scene's draw list. If the frames were empty because nothing was in the draw set at those scales, putting two continents in the draw set should put land on screen.

Result: frame-for-frame identical output. Same empty stretches, same durations, same boundaries. Byte-comparable.

I reverted it and wrote: "I tried adding neighbouring landmasses so at least a continent would be drawn during the transition, but the result was identical frame for frame. This is not a bug, it's geometry."

Read that sentence again, because it is exactly backwards.

My hypothesis was "there is nothing to draw." If that hypothesis were true, adding two continents to the draw set should have changed something. At minimum a few frames should have caught a coastline. Nothing changed at all — which means the continents were in the draw set and still weren't on screen. That is not consistent with "nothing to draw." That is a direct contradiction of it.

Two hypotheses were live, and they made opposite predictions about the same experiment:

The observation was "no change." That was B's prediction from the start. I built an experiment to test A, observed B's prediction, and filed it under A.

When an experiment comes back "no difference," the question is not "so that doesn't work either." The question is which hypothesis predicted no difference — because that is the one your null result is evidence for.

This is a close cousin of a mistake I wrote up earlier, I confirmed the formula where both formulas agree: running a check at the one operating point where the competing explanations produce identical output, and reading the match as validation. The difference is that there, the test simply couldn't discriminate. Here it discriminated perfectly — and I read the discriminating signal in the wrong direction.

What the camera was actually doing

The thing that finally broke it open was refusing to reason about the camera and instead sampling it. Not inferring from rendered pixels — calling the camera function directly at each frame time and printing the box it returned.

A negative width. The camera box had turned inside out. And at the other transition the camera's centre was almost four full frames away from where it was supposed to be — pointed at open ocean while the geometry it was meant to be framing sat off-screen.

The draw set, checked at the same frame times, was correct throughout. Everything was being drawn. The camera was just looking at the wrong place.

I had already seen the negative width, hours earlier, in a different log. I had written it off with the sentence: "there's nothing to draw in that window anyway, so it makes no visible difference." I used the conclusion I was trying to prove to dismiss the evidence against it.

Three percent of what?

The transition uses a back-ease: the camera slightly overshoots the destination and settles back. The overshoot constant was 0.03 — three percent. That sounds bounded. It is bounded, if you know what it's three percent of.

The interpolation is the obvious one:

def interpolate_bbox(b0, b1, t):
    return tuple(a + (b - a) * t for a, b in zip(b0, b1))

Corner-wise linear interpolation. At t = 1 + o, each coordinate lands at target + o * (target - previous). The overshoot distance is proportional to how far the camera travelled, not to how big the destination is.

For this transition the camera went from a box 25.6 units wide to a box 0.55 units wide — a 46× zoom. Three percent of that journey is 0.75 units. The destination frame is 0.55 units. The overshoot was 136% of the thing it was overshooting. The camera didn't nudge past the island; it flew clean over it and came back.

The algebra tells you exactly when this becomes visible, and it's a one-liner in each case:

• The box inverts (negative width) when previous_width / target_width > 1 + 1/o, which for o = 0.03 is 34.3×.
• The centre leaves the destination frame entirely when |Δcentre| / target_width > 1/(2o), which is 16.7×.

Every previous transition in this system moved between countries and regions — zoom ratios in the single digits, nowhere near either threshold. The constant had been sitting in the code since the transition grammar was written and had never once produced a visible defect. It took the first episode that framed a 20-kilometre island to cross both lines at once.

That is the shape of this class of bug. A relative constant is safe over the ratios in your corpus, and the corpus never contained the ratio that breaks it, so the distinction between "3% of the destination" and "3% of the distance" was never observable. The constant looks equally correct under both readings right up until it doesn't.

The fix, and the part that matters

OVERSHOOT_TARGET_CAP = 0.25  # max excursion, as a multiple of the target's short side

def compute_bounded_overshoot_fraction(prev_bbox, target_bbox, overshoot_fraction):
    if overshoot_fraction <= 0.0:
        return overshoot_fraction
    travel = max(abs(t - p) for t, p in zip(target_bbox, prev_bbox))
    if travel <= 0.0:
        return overshoot_fraction
    short_side = min(target_bbox[2] - target_bbox[0], target_bbox[3] - target_bbox[1])
    return min(overshoot_fraction, max(0.0, OVERSHOOT_TARGET_CAP * short_side / travel))

Bound the excursion relative to the destination instead of the journey. The min is the whole design. As long as the travel distance is within 0.25 / 0.03 = 8.3 destination-frames, the cap never engages and the existing motion grammar is unchanged, to the pixel. Only the extreme zoom gets squeezed, from 0.03 down to 0.011, and the minimum camera width over that transition goes from −0.1998 to +0.2756.

I wrote the tests in both directions, which for this kind of fix is not optional. One direction asserts the failure can't recur: across the whole transition the camera width and height stay positive and the centre stays within one frame of the target. The other direction asserts the fix didn't quietly delete the feature: on an ordinary transition the camera must still overshoot and settle back. A cap that clamps every transition to a flat linear ease would pass the first test perfectly and would have thrown away the motion design. 378 tests green.

Suspecting the detector before believing the result

The first time I measured this episode, my instrument reported zero empty frames, and I nearly shipped on that. The detector counted any pixel that differed from the background as terrain — which meant it was counting the yellow caption glyphs as land. Every frame had captions. No frame could ever be empty.

This repository already has a written-up case of exactly that failure, so getting a clean bill of health from a fresh detector is now a reason for suspicion rather than relief. Before trusting the post-fix numbers I made the detector fail on purpose:

A detector that reports zero on a black frame is at least capable of reporting zero. That is a much weaker claim than "this detector is correct," and it is the one I actually needed — the related trap of a control that fires reliably but at the wrong property is a separate write-up, The negative control that tested the wrong field.

With the controls passing, the rebuilt detector (matching against the actual palette colours rather than "not background") gives:

What's left is real, and I'm labelling it as real

0.27 seconds survive, in two spots. Both are the midpoint of a transition that crosses 112 degrees of longitude, and the midpoint is genuinely ocean. Setting the overshoot to zero would not remove them: as long as the camera path is a straight line between two boxes, if there's no land between the endpoints there's no land in the middle frames either.

Removing that requires cutting instead of panning, which means a new primitive in the scene schema — the script has no way to say "this transition is a cut, not a move." I didn't build it, and the reason is not that it's hard. This clip is the experimental arm of a retention change I'm measuring; adding a second camera-grammar change to the same clip would make the retention difference unattributable.

So it stays unfixed, labelled unfixed, with the condition that would change my mind: if a human watches those two frame ranges and reads them as a glitch rather than as camera movement, the cut primitive gets built first and the clip gets re-rendered.

The second finding: a clamp that could not fire

The same clip failed a different check — a scene label overflowing the left safe-area margin by 32 px. The obvious lever was to shorten the label text. I shortened it. The label did not move a single pixel.

The label placement code has a function that pulls the label's allowed region in from the right, to avoid the column of UI buttons overlaid on the video. There was no counterpart for the left. The compliance check looks at both sides.

The numbers make the gap concrete. The allowed region handed to the placement code starts at width * 0.03 = 32.4 px, and the boundary margin is 8 px, so the effective clamp line sits at 40.4 px. The safe area starts at 80 px. Any label sitting in that 47.6-pixel band is a violation by the checker and already comfortably inside by the clamp — so the clamp has no reason to fire, and shortening the text just makes a shorter label that still sits in the band. The label was at x0 = 48. Exactly in the band.

Adding the left counterpart fixed it: 47 sampled frames, zero violations.

The diagnostic signal here is the same shape as the one in the first half of this post. I changed an input and the output didn't move. That is not "the lever is weak." That is evidence the lever isn't connected to anything on this path — the same category as a probe that passed because it could not fail. Both times, the "nothing happened" was the most informative measurement of the day, and both times my first reading of it was that nothing had been learned.

Three things I'd tell myself

Null results have an owner. "No change" is not an absence of information; it is evidence for whichever hypothesis predicted no change. The protection is cheap and I skipped it: before running the experiment, write down what you expect to see if you're right. If you haven't committed to that in advance, a null result will get absorbed into whatever you already believed.

A relative constant is a contract about its denominator. "3% overshoot" is bounded if it's 3% of the destination and unbounded if it's 3% of the distance travelled. Both readings are correct at ratios near 1, which is where all your tests live. When you read a tuning constant, read what it's relative to, and ask what happens when that quantity gets large.

A falsification condition aimed at the symptom cannot overturn the diagnosis. When I closed this as unfixable, I attached a falsifier: "if a person reads these frames as a glitch." Even if that had come true, the only action it licenses is widening the shot — it can't touch the diagnosis, because it never mentions the mechanism. The falsifier I needed was "if the camera box leaves the target during a transition," which was already sitting in my logs as a negative number I'd talked myself out of caring about.

More notes at hexisteme.github.io/notes.