Six Phases: Diagnosing an Ad Generator's Regression Against Its Own Corpus

"Reliability went up and intent went down." That's the one-line summary of a regression I found by comparing two months of a video generator's output against itself: July's runs against August's, 295 scenes against 268. Every individual change made between them had shipped for a defensible reason. The aggregate was a pipeline that completed more often and made a worse ad every time. Fixing it took six phases, each one measured against the corpus rather than against a guess at what "better" meant.

A regression nobody can point to a single bad commit for is still a regression. Measure the aggregate, not the diffs.

The measurement that started it

Phase 0 (PR #306) is the diagnosis, not a fix:

July made long, product-featuring videos that often crashed mid-render. August made short, product-free videos that reliably completed. Every fix made in between had chased reliability — and had done it, one at a time, by narrowing what the planner was willing to attempt until the safest possible output was also the least differentiated one. Phase 0's fix isn't a new mechanism; it's restoring the specific behaviors the corpus proves the best July runs had, verified line by line rather than reconstructed from memory of what "used to work."

Phase 1: the right entity, not just a product

PR #307 closed a gap the product's own spec had flagged as a correctness bug: the script decides which brand entity a beat highlights — often a mascot character, not the physical product — and the live planner had no concept of "mascot" as a distinct, castable entity. One reference ad opens on a chip-mascot character. The reference's own expected-output clip opens on the same mascot. The pipeline's render opened on a product case with a face crudely implied, because a case photo was the only visual anchor a mascot casting decision had ever had available. The fix is a real entity catalog — case, buds, chip, mascot — each with its own established path from script slug to committed asset, so "this beat is about the mascot" is a decision the planner can actually represent instead of approximating with the nearest product photo.

Phase 2: a mascot that performs instead of a product that sits still

PR #308 diagnosed the specific viewer complaint of "we lost the hook" as a branch-selection bug: a mascot span was falling through into the same code path as a plain product shot, and got the product shot's own animation direction — explicitly, "no people, hands, fingers, or figures anywhere, just the product in its setting" — applied to a character that was supposed to be performing. The result was a static figurine on a blank background for nearly five seconds at the top of a fourteen-second ad, while the reference it was meant to echo had its mascot animated over dynamic imagery for the same beat. The planned fix was a new dedicated role for character spans; the shipped fix turned out simpler once written, because the existing rendering primitives already supported it — the branch just needed to route there.

Phase 3: a b-roll bank that was already built and switched off

PR #309 is the phase I'd flag as a reminder to check configuration before writing code: a Drive-backed b-roll library, with a tagged index carrying duration, aspect ratio, and audio metadata, streamed to the player and cached for FFmpeg, already existed and was already wired into span selection. It simply no-op'd because one environment variable — the folder ID it reads from — was unset. The documentation describing the system's current state said "none; b-roll is generated from scratch," which had been accurate once and had gone stale without anyone updating it after the bank was built. The genuinely new work in this phase was smaller: when nothing in the bank matched a scene well enough to use outright, the closest available clip was being discarded rather than used as the best available answer to "what fills this beat" — a completeness bug sitting next to a configuration bug.

Phase 4: the pace guess was wrong by a measured 11%

PR #310 traces the entire dead-air and time-compression bug family back to one shared root: every scene's target clip length was computed from a words-per-second constant of 2.5 — and the pipeline's own code comment, sitting right next to that constant, recorded the actually-measured delivery rate of the voice model in use as 2.78 words per second. Every scene was sized roughly 11% longer than the voiceover would actually take to speak, so the picture consistently outlasted the line and the video model filled the gap with idle behavior — dead air. The new default, 2.75, is deliberately still just under the measured 2.78: erring slightly long is recoverable at the stitch step by trimming the tail; erring short cuts a line off mid-word, which isn't recoverable at all.

Phase 5: a share is not the quantity that matters

PR #311 unwinds an assumption baked into a single constant: MAX_BROLL_SHARE = 0.5 treats a 15-second ad exactly like a 45-second one, because it caps a percentage of runtime rather than the actual viewing experience that percentage produces. A 45-second reference ad spending 45% of its runtime on cutaways spreads that across roughly fifteen separate shots — each cutaway reads as brief texture between longer presenter shots. A 14.5-second live render spending the identical 45% spent it as two blocks of roughly 3.5 seconds each, alternating presenter, mascot, presenter, mascot — the same math producing a viewing experience that reads as choppy back-and-forth rather than texture. The fix stops treating the share as the governing quantity and instead governs the number and length of individual interruptions directly, because that's the variable that actually determines how a cut sequence feels.

The pattern: measure against your own corpus before trusting your own instinct

None of these six phases required a new model, a new vendor, or a new architectural layer. Every fix was available in the codebase or the documentation already — an unset environment variable, a branch that fell through to the wrong case, a constant that had never been updated after the number it was based on was independently measured and written down two lines away. What made all six findable in one week rather than continuing to accumulate as separate, unconnected complaints was Phase 0: stopping to measure the aggregate output against a real corpus of past runs, instead of triaging each new complaint as its own isolated bug.