The Check That Cannot Fire: Absolute Thresholds Silently Disable Your QC

Originally published on hexisteme notes.

I built a quality-control harness to check rendered video deliverables — six axes covering geometry, audio lineage, edit-point clicks, edit-point silence, frame lineage, and segment boundaries. I ran it against ten synthetic surrogate files with deliberately injected defects. Thirteen cases, all green. Then I ran it against actual lecture footage and five cases broke.

The interesting part is not that it broke. It is that all three root causes were the same shape, and that shape is one I had already been bitten by twice before in the same repository without recognizing it as a category.

A threshold written as an absolute constant is a hidden assumption about the scale of your input. On content whose scale differs from your fixtures, that check does not fail — it stops firing. And a check that stops firing reports success, which is why it can live in your suite forever.

Thirteen green fixtures, five real failures

The harness is three layers: a pure decision layer with no I/O, an executor that shells out to ffprobe/ffmpeg, and a verification harness that runs positive and negative controls. The synthetic surrogates were files I generated with known defects — a 200 ms audio delay, a dropped frame, a shifted segment — so I could assert that each axis fires on the defect it owns and stays quiet on the others.

Every one of those passed. Then real footage: quiet lecture audio, a static presenter frame, ordinary re-encode noise. Five failures, three distinct causes.

Cause A — an absolute peak threshold on a click detector

The edit-point click detector looked for a discontinuity in the waveform at each junction:

# before — 0.12 is a number that assumes some particular audio
fired = [s for s in stats if s.peak_step > 0.12]

Measured on the real file: the peak step at control points — deliberately chosen non-junction locations — had a 95th percentile of 1.43e−2. The synthetic click I injected to test sensitivity peaked at 2.00e−2. The injected defect was 1.4x the background and unmistakable in the data, and the threshold of 0.12 swallowed both of them. On quiet content this detector was structurally incapable of firing.

The fix is not a smaller constant. It is to stop using a constant:

# after — compare against local background measured in the same run
ratio = peak / max(ctx_p99, ctx_floor)
ratio_threshold = max(k * control_ratio_p95, min_ratio)  # threshold derived from control sites
fired = [s for s in stats
         if s.ratio > ratio_threshold and s.peak_step > peak_floor]

On the same data this separates a clean junction at 0.084 from an injected one at 2.8e+3. Four orders of magnitude, from a detector that minutes earlier could not distinguish either from zero.

Cause B — an absolute margin on frame comparison

The frame-lineage axis has to tell "this output frame came from that source frame" apart from "it came from the one two frames over." I had encoded "distinguishable" as min_margin = 0.002.

When I actually measured it, the signal was 34 to 45 times the re-encode noise. It was not marginal. The constant was eating a clean signal.

# after — measure this run's re-encode noise, use a multiple of it
noise = median(1 - best_ncc_per_frame)          # noise floor for THIS run
detect_threshold = k_noise * max(noise, floor)  # k_noise = 4
detectable = content_margin > detect_threshold  # is the content separable at all?

That last line matters as much as the threshold. Before asking "did the check pass," ask "on this content, could it have failed?" If the answer is no — a static frame where every candidate correlates at 0.9999 — the honest output is not pass. It is discrimination_absent.

Cause C — an undocumented applicability limit

The third one was not a constant, but it belongs to the same family. The edit-point silence check works only on silence runs between 90 and 359 ms, and it cannot see a 200 ms delay defect at all. None of that was written down anywhere. A check whose applicability limits are unrecorded produces a pass that nobody can interpret.

Proving the fix tightened rather than loosened

Here is the obvious objection to everything above: you changed thresholds until the failures went away. That objection is correct by default and has to be answered with a measurement, not an assurance.

The measurement is a detection floor: inside the same run, on the same content, inject a synthetic defect at a sweep of amplitudes and record the smallest one that still fires the check.

Fifty times more sensitive with every prior positive control still green. That is the difference between loosening a check and turning one on. Without the floor number I would have had a story; with it I have a comparison.

What the working harness then found

The point of all this was to gate real deliveries, so: first live render, 10,630 frames, 48 seconds, geometry axis perfect — exact frame count, exact sample count, correct codec, zero padding. And the audio-lineage axis failed.

The last 29.3 ms of the delivered file was full-scale white noise. Peak 0 dBFS on both channels, zero-crossing rate 0.481, spectral flatness 0.850 — the signature of white noise (0.5 and 1.0 respectively), where speech sits near 0.05–0.15. It started 192 samples into the final frame, so it was not aligned to any frame boundary and therefore not an edit event. Two full renders produced a byte-identical result, down to the same QC report digest; a third, shorter render over just the tail reproduced the same burst sample for sample. Not a glitch — a deterministic defect.

Two things about that are worth separating. First, everything upstream passed. The editor's own API cheerfully reported a correct timeline, because "what the tool remembers" and "what landed in the file" are different facts and only one of them is the deliverable. Second, the surviving-hypothesis work was only possible because the harness could actually fire.

Localizing it took eight experiments, and six hypotheses died:

• Render-tail artifact — dead. Rendering a range that ends one frame earlier produces a clean tail.
• Contaminated source media — dead. The source measures −22.55 dB there, and a file assembled from the same plan with ffmpeg is bit-identical to the source.
• Renderer reaching the end of the timeline — dead. I appended a silence handle so the frame in question was no longer last, rendered through it, and the new final frame was digital silence while the burst stayed exactly where it was.
• "The last frame of any audio clip" — dead, via a clip that consumed all of its media.
• "The clip ends before its media does" — dead, via a clip that deliberately left 15 frames unused and came out clean.
• "The last placed frame of this particular clip" — dead. I placed the same media at a completely different point on the timeline and the corruption moved with it, staying at the same offset inside the media.

That last one converted the problem. The defect was never bound to timeline geometry; it was bound to a fixed position in the media file. Measured precisely, the burst spans source samples 17,006,592–17,008,031 — 1,440 samples straddling 17,008,000, which is exactly where the baked program audio ends and the trailing digital silence begins. The original "1,408 samples" I first measured was the same burst, clipped by the render ending.

The clean control was already sitting in the project: a second audio bed whose length happened to be a multiple of three frames, so it needed no trailing padding. Its final frames render clean, at exactly the reference tail level. Container layout matches in every respect I could measure — uniform 1024-sample packets, one partial packet at the end of each, no edit lists, no stream discontinuity. The one difference that survives measurement is that the failing file ends with 3,200 samples of exact digital zero and the clean one ends with zero of them, in real room tone.

So I changed the padding to continue the real source audio instead of appending silence, and shipped it. The final render is clean: zero burst frames, a last-frame peak of 0.00156 where the same frame had been 0.531 RMS, and a tail level of −22.554864 dB — matching the clean control to six decimal places. Not quieter. The same.

Before deploying the change I gave it a check of its own — count the trailing zero-valued samples in the baked carrier and fail if there are any — and I ran that check against the old file and the control first. Old file: 3,200, fails. Control: 0, passes. Only then did it go in. A check I had not seen fail would have been the fourth instance of this essay's defect, added by the fix for the third.

One more thing happened that is worth the paragraph. I rebaked the file in place, re-rendered, and the burst was still there. The QC digest came back byte-identical to the previous run — not a similar failure, the same output. The renderer had never read the new file. Its media cache is keyed on path and is not invalidated by mtime; the fix was on disk and invisible at the same time. Note what the same signature meant in two places: earlier, two byte-identical renders were my evidence that the defect was deterministic; here, a byte-identical digest was evidence that nothing had been re-read. Same number, opposite conclusion, and only the surrounding experiment tells you which. Versioning the filename and relinking fixed it. Had I been comparing screenshots, or listening, I would have concluded the prescription was wrong and gone hunting for a mechanism that did not exist.

Then I closed the last open assumption, and it bit me one more time. Every measurement so far had gone through one decoder, so I re-decoded the same files with a second, unrelated one — Apple's CoreAudio instead of ffmpeg — after first confirming on the old file that the second decoder could see the burst at all. It could. The two decoders then came out bit-identical, max absolute difference 0.000e+00.

But they disagreed on the number I had been quoting. My tool reported a last-frame peak of 0.0022; CoreAudio said 0.00156. The ratio is 1.41421 — √2 — and the cause was mine: my tool asked ffmpeg for a mono downmix, and ffmpeg's stereo→mono is energy-preserving (1/√2 per channel), so on this dual-mono file it returned √2 × the actual sample values. Every absolute amplitude that tool had ever reported was 3 dB high, including two numbers earlier in this essay, which I have corrected. The comparisons all survived — a uniform gain cancels in a ratio, which is exactly why the ratio-based rewrite was the right fix — but the absolute figures were never the file's figures. The tool even printed a per-channel astats cross-check right next to the inflated numbers, 3 dB apart, and called itself cross-checked.

The same failure with no code in it

While writing this up I found the identical shape in a protocol rather than a program. I have a standing rule that says: when a general engineering lesson comes out of a project, proactively offer it as a writing candidate. For two months I believed the rule was working. It recorded nothing.

"Zero offers made" and "no material worth offering" are different states, and the rule had no instrument that could distinguish them. Every month it silently reported success. It is the same defect as peak_step > 0.12 with none of the arithmetic — a check with no demonstrated way to fail.

What transfers

1. Derive thresholds from signal-to-noise measured in the same run. A constant encodes your fixture's scale. The moment real content differs, the check switches off quietly, and quiet failures are indistinguishable from success.
2. Passing synthetic fixtures is not evidence of sensitivity. Fixtures contain the defects their author already imagined. Evidence of sensitivity is a detection floor measured in-run.
3. Every check needs a demonstrated way to fail. If you cannot show it failing, its passing carries no information. This was the third instance in one repository — the first was a fade-amplitude gate that was tautological because the cuts already snapped to silence; the second was a waveform check whose own prep procedure made passing the only possible outcome, which read a plain split as a surviving crossfade. Those two were checks that could not discriminate. This one is a check that could, until the content changed scale.
4. When you cannot discriminate, report that — do not invent a threshold. pass and could not measure are different facts, and merging them means the next person reads the second as the first.
5. Write down each check's applicability limits next to its output. "Cannot fire on inputs of this shape" is part of what a pass means.
6. Any absolute number you report needs a second, independent path to produce it. Ratios protect your logic; they do not protect your numbers. An instrument that applies a hidden transform stays internally consistent forever — every comparison it makes is still correct — and nothing inside the system can tell you. Only a differently-built measurement can.

Where this could be wrong

• The ratio-based thresholds are floored by two constants (ctx_floor, peak_floor) to keep them from exploding where the local noise approaches zero — genuine digital silence. Those floors are themselves absolute constants, which is the exact pattern this piece is about. Unresolved; currently watched by the detection floor, which would degrade if a floor started swallowing content.
• A detection floor measured with one injection type (synthetic clicks) is weak evidence for defects of other types. The real defect here was a white-noise burst, not a click, and it was caught — that cross-type hit is the only evidence I have that the floor generalizes, and one data point is one data point.
• The clean control differed from the failing file in more than the trailing zeros: different segmentation, different length. The matched comparison — the same file re-baked with only the padding changed — is what closed it, and it came back clean. But that is one file, one renderer version, one machine. I have a trigger and a prescription that works; I do not have the mechanism inside the closed-source renderer, and I cannot tell you what else shares it.
• The decoder assumption is closed — two independent decoders agree bit-for-bit — but closing it revealed a 3 dB scale error in my own reporting, which had gone unnoticed through the entire investigation because every comparison was internally consistent. I do not know what else in this harness is internally consistent and absolutely wrong; a second independent path is the only thing that found this one, and I have exactly one.

More notes at hexisteme.github.io/notes.