A planner that generates a full shot list for an AI ad can be wrong in two very different ways. It can produce a bad plan — and a human or a downstream check catches it. Or it can produce a plan that is internally inconsistent, or a scoring step that never actually runs, or a review gate that reports a healthy number while the real number is something else entirely — and nothing catches it, because every individual system reported success. Across roughly fifteen PRs on the main video-generation service, I went looking specifically for the second kind: failures that pass their own checks.
The dangerous bug isn't the one that throws. It's the one where every component reports success and the composite is still wrong.
A scorer that generated candidates and never looked at them
PR #294 found that the frame-generation step created several candidate frames per scene — configurable, paid for individually — and then hardcoded chosenCandidateIndex: 0. Every alternative was generated, paid for, mirrored to storage, and discarded unlooked-at. That's the "generate divergently, converge by scoring" pattern from the research literature with the scoring half simply missing. The economics made the gap concrete: on a completed run, a full clip costs $1.55 and an extra candidate frame costs $0.05 — the pipeline was already paying for the option to pick the best frame and then not exercising it.
PR #299 found why the fix from #294 didn't show up in production output at all: both fan-out sites that would call the scorer gated on !job.autopilot, and autopilot is the production path. The scorer only ever ran when a human was manually driving the review gates — which is how it had been tested, which is exactly why the gap wasn't obvious. The gating condition had been correct when originally written; both halves of the reason had since changed underneath it, and nobody had gone back to check whether the guard still made sense.
A review gate that reported 0% for a run that spent 43%
PR #293 is the one I'd call the most dangerous in this batch, because of what it would have let ship unnoticed. The log line at the script-review gate is titled "reference split by seconds" but was computed entirely from the plan side rather than the reference — on one run it printed sharePct: 0, cutawayCuts: 0, sources: 0. The same run then rendered three cutaways over 43% of its runtime from two sources. An operator reading the gate before approving the run would see a report of essentially no b-roll and approve a run that actually had extensive b-roll — the review gate's entire purpose, catching a run before it ships, defeated by a metric computed from the wrong side of the comparison. This is also the PR that made the two before it, #290 and #291, trustworthy: without a correct gate reading, both of those fixes could have shipped a regression that nothing downstream would have flagged.
A plan that contradicts itself, caught at the gate instead of the render
PR #282 traced a real failed run back to its own plan data. One scene's motion field explicitly read "no person on screen," while the same scene's role was presenter and its productState was none. The frame-generation step then attached a person reference and zero product references and rendered a person holding nothing, in a scene the plan itself said shouldn't have a person in it. A second scene independently declared "no talking face" while also being typed as a presenter scene. Every fact needed to catch both contradictions was already present in the plan — nothing needed to be inferred or predicted. The fix adds a gate that gets an internally-inconsistent plan is caught before a paid render, not after one produces visibly wrong footage.
Measurement code nothing calls
PR #305 is a direct, deliberate audit: for every scoring or measurement function in the codebase, does anything actually read its output? Two full modules — one comparing shot lengths against a reference, one computing an energy curve over the audio — exported five functions each, all imported by the same orchestration file, and called zero times. They looked like active infrastructure from an import graph. They were pure overhead: computed, discarded, never read. The same PR made the frame scorer from #294 able to actually disagree with the default choice, closing a gap where the scorer existed, ran, and always happened to agree with candidate zero regardless of what it measured — which is a scorer in name only.
A dead pipeline, kept alive by nothing but its own existence
PR #275, labeled "D2," deleted an entire older scene-swap and one-shot-UGC pipeline outright, and PR #276 removed a second plan shape — ugcVariant — that PR #274 had already established nothing could actually request. Both deletions followed the audit discipline from #305: not "this looks unused," but "I checked every call site and there are none." PR #286 dropped two job-state fields nothing could write to anymore, for the same reason. Deleting confirmed-dead code isn't cleanup for its own sake here — every line of dead code left in a planner this size is a future false positive waiting for someone to assume it's load-bearing.
A cap that outlived the reason it existed
PR #292 found a scene cap that had made sense under an old architecture — when a fast-cut reference video meant one generated clip per detected shot, the cap bounded runaway cost — and had kept exactly the same numeric value long after the function computing it changed to have exactly one caller, whose output now only grounds transition boundaries rather than driving clip count. The same constant, unchanged, had silently gone from "a real cost control" to "a cap that deletes evidence the vision pass already gathered," because nobody had revisited why the number was what it was after the code around it changed.
Diagnosing instead of guessing at a vendor 422
PR #298 responded to a Kling failure with a message that named the wrong cause — "video duration is not supported," on a request asking for exactly the duration the vendor's own documentation allowed. Reading back the actual checkpoint data showed the error was inconsistent between otherwise-identical requests, which meant the message was misleading, not descriptive. Rather than patch around the reported symptom, the fix logs the literal request body sent on rejection and stops resubmitting an already-rejected payload verbatim. That logging is what let PR #303 resolve the real mystery two days later on its first occurrence: a 10MB payload size limit, triggered by base64 encoding inflating an image by 4/3 past the vendor's raw-byte ceiling — a bug that four separate paid diagnostic probes, at roughly six dollars total, hadn't been able to pin down until the request itself was actually logged.
Never author over a legal disclosure
PR #297 is the one correctness bug in this batch with a compliance dimension, and it was caught before it shipped rather than after: checking a new overlay classifier against real corpus data, before running the validation render it was built for. Every measured overlay on one reference ad was a "compensated for testimonial" disclaimer, and the classifier scored all of them as ordinary graphic beats to mirror — which would have filled each one with a line from the pipeline's own marketing script, replacing a required legal disclosure with a sales claim in the exact screen position a viewer expects the disclosure to appear, on an ad for a medical device. The fix is a hard classification rule carved out ahead of the general classifier, not a tuning adjustment to it.
The pattern: check whether it fires, not just whether it's correct
Nearly every fix in this cluster was correct code sitting behind a condition that silently prevented it from running, or a metric computed from data that looked plausible but was measuring the wrong side of the comparison. Unit tests on the scorer, the classifier, and the review-gate calculation would all have passed in isolation — they were internally correct. The bugs only existed at the seam between components, which is exactly where unit tests don't look and where "does anything call this in production" has to be asked explicitly, on purpose, as its own audit step.
