Nobody on this project has vendor dashboard access. That single operational fact, stated plainly in more than one PR description, is the reason a cost ledger on the main video-generation service went from "probably fine" to actually verified — and the reason verifying it took longer than writing it. You cannot check a number against a source of truth you don't have. You can only check it against itself, from a different angle, and see whether the two angles agree.
A measurement script that breaks does not fail loudly. It publishes a wrong number, and a wrong number that looks plausible is worse than no number at all.
Six defects, one shape: reads correctly, reports success, does nothing
The ledger audit (PR #360) found six separate billing defects, and every one had the same signature: code that read as correct, compiled, and produced a log line indicating success — while the actual charge silently vanished or landed wrong. A billing idempotency note was keyed from a module-level counter that reset to zero on every redeploy, while the note history itself survived in a Drive snapshot; Railway restarts mid-run routinely, so a resumed run's first metered call regenerated a note that already existed, and the dedup logic dropped a charge the vendor had genuinely billed. A checkpoint-invalidation function and the ledger's own deduplication were treated as one fact when they were two — invalidating a checkpoint said "this artifact no longer exists," which triggered a real re-render and a real vendor charge, while the ledger dedup logic, unaware anything had changed, kept believing the old note still covered it. Every editor click of "fix this scene" was a real $1.26–$2.10 charge landing at $0 in the books.
The fix for the second bug is worth pausing on, because it's a defensive pattern in its own right: rather than trust fourteen call sites to remember to drop the stale cost note when invalidating a checkpoint, the drop was moved inside the invalidation function itself. A caller that forgets doesn't fail loudly — it silently bills at zero, which is the one failure mode a fourteen-site checklist can never fully close.
A resume path that re-decided the render instead of resuming it
The same PR found a second class of bug hiding in the resume path. restoreFromFolder rebuilds a job by hand after a restart, and it omitted four fields — the run's settings, its engine choice, its aspect ratio, and an image-model pin. Those weren't left unset. A downstream function re-pins anything absent, using whatever the current environment happens to be — so a run that started at 9:16 could finish at 16:9, or silently change rendering engines mid-render, purely because it happened to restart. The bug isn't visible as an error anywhere; it's visible only as a delivered video that doesn't match the settings the editor originally chose, discovered well after the run has been paid for.
Building the check that could actually run
PR #372 responds to an impossible verification instruction: a prior handoff had named "compare the ledger against the fal/OpenAI/ElevenLabs dashboards" as the only proof the wave-1 fixes worked. Nobody has that access. An unprovable check is not a check — it's a line item that never gets ticked. The alternative built here, reconcile-ledger.mjs, recounts every paid artifact from the job's own persisted state, prices each one using the codebase's own rate constants, and compares that expected range against what the ledger actually recorded:
clips rendered 15 | clips charged 15 (9 scene + 6 span)
frames 31 | clip-critic 48 | tts 1004 chars
expected $13.82 .. $18.21 ledger reports $17.98 VERDICT: CONSISTENT
That doesn't prove the vendor's per-second price matches the code's constants — no dashboard access means that gap stays open, and the script says so honestly in its own output rather than burying the caveat. What it does prove is narrower and still real: no charge silently dropped, and no charge silently missing. The PR also caught its own near-false-positive before shipping: a cutaway scene has no per-scene billing note at all, because its picture is billed under a b-roll span note instead — six consecutive scenes on one real job that look unbilled and aren't. Checking that distinction before raising it as a bug is, as the PR puts it, the difference between writing a tool and reporting an incident.
The report that disagreed with its own ledger
PR #382 replayed an actual paid production run through an improved plan-gate and got eleven errors on a plan that had originally shipped with zero — sixteen warnings, autopilot-approved, and three of the eleven were defects a downstream frame critic had already reported, after the clips were purchased. The plan's own acceptance criteria described a closed white oval case; the critic that ran afterward reported no case visible at all, only a ceramic mug. The gate hadn't been wired to check what the critic was already catching on its own, one stage later and at full clip cost.
The same replay found a second, sharper bug: every one of fifteen billing events for a completed render was priced at exactly $0.14 per second, cleanly matching one vendor tier's rate — on a run whose actual ledger had charged the other tier's rate, $0.112 per second. The root cause was almost embarrassingly simple once found: the function that recorded the cost passed a full pricing specification, and the function that emitted the reporting event passed the bare string "pro". Two code paths deriving the same fact, one of them wrong, and nobody had noticed because the wrong number was still a plausible-looking number. As the PR notes about the fix: expect reported totals to rise with no change in real spend. That's the fix working, not a regression — which is exactly the kind of sentence that needs to be said explicitly, because a number going up looks like bad news until you know why.
A reconciler with four bugs of its own
PR #389 ran the new reconciliation script against all three paid production runs instead of just the newest one — and found the reconciler itself had four defects, discoverable only by testing it against more than a single lucky case. It had no concept of a multi-shot bundle, so a bundled group's members carried no individual billing note and the script reported "3 rendered clips carry no charge" on a ledger that was actually fine — a false alarm that had been silently wrong on every bundled run since the feature shipped. It attributed bundle membership by parsing a note's name rather than reading the actual persisted group data, which happened to balance by coincidence on one run while misattributing six scenes on another. It summed two separate ledgers together, reporting "15 rendered, 21 charged" in a way that read as overbilling when it was actually scene coverage plus separate b-roll asset charges added together. And it omitted two whole cost categories — voice cloning and QA passes — which on one run accounted for $0.71 of what looked like a $1.17 unexplained gap.
All three runs reconciled cleanly once fixed: $16.34 against an expected $14.21–$17.46, $17.98 against $15.50–$19.05, $17.26 against $14.20–$16.80. The PR also caught a false positive it had just introduced, in the same pass, the same way it caught everything else — by running the check against a fourth case rather than trusting the fix.
A $4.43 sweep that couldn't verify the one thing it existed to check
PR #383 added pin fields recording exactly what a run had been configured with — which feature flags actually resolved to true after their dependency logic ran, which TTS model and its measured delivery rate, which rendering engine, which aspect ratio — because "flags confirmed over the wire" had been an unfalsifiable claim with nothing in the job's persisted state to check it against. PR #394 then ran a cheap $4.43 diagnostic sweep specifically to exercise those new fields, and found every one of them absent on all six references swept. The cause: a wrapper function that acquires all five pins was documented as sitting on every path that continues a job, and mostly did — except the very first step of a brand-new run called a lower-level function directly, skipping the wrapper entirely. A job only acquired its pins once an editor approved a gate and frame work began. The one claim the pins existed to make checkable — what a plan-gate run was actually configured with — was, by construction, the one thing a plan-gate run could never prove about itself.
Measuring the wrong build
PR #396 and #399 are a two-part story about a deploy-verification script that looked correct and measured the wrong thing. A deploy-wait script reported "STABLE" fifteen seconds after a merge — but stable can mean "the previous build settled down," not "the new build arrived." A merge landed, the wait script returned success, and an $0.82 gate run measured the previous commit; the error count dropped from nine to five, which read as "the fix helped but didn't finish," when the fix in question hadn't deployed at all. Root-causing that led to #399: Railway only exposes its commit-SHA environment variable to a Docker build if the Dockerfile explicitly declares it as a build argument, and this one didn't — so every production build ID had silently been a millisecond timestamp instead of a commit hash, and the health endpoint could report that a deploy happened without being able to say which code was actually running. "Which code is running" had been an inference the whole time, and that inference cost real diagnostic dollars to discover was wrong.
Spending $14 unattended on a plan already known to be broken
PR #435 closes the loop on the whole cluster with a policy decision as much as a code fix. A semantic plan-QA check had been hard-coded to always report as a non-blocking warning, on the reasoning that one LLM's opinion shouldn't be allowed to halt a run outright — and on a real render, that check had caught a genuine defect for free, at the plan-gate stage, before a cent of clip spend: a beat's picture planned a slow, static drift under a line that said "instantly." The pipeline logged the warning and rendered anyway, for roughly fourteen dollars, and the same contradiction showed up in the editorial review of the finished video. The fix isn't to make every warning blocking — a gate that halts on every advisory opinion is a gate editors learn to click past. It's to promote specifically this one class of internally-contradictory finding to a hold rather than a hard failure: the run parks at a gate it was already going to pass through, a human approves or fixes the plan, and nothing is discarded. A held run now costs about twenty-five cents instead of fourteen dollars, for the exact same finding.
The pattern: verify with what you actually have
Every fix in this cluster shares a structural constraint: no vendor dashboard access, no ground truth to check the ledger against directly. The response wasn't to give up on verification — it was to build verification out of whatever is available: the job's own persisted state, the code's own rate constants, a second run to test the first run's tool against, a replay of an already-paid job through an improved gate. None of these prove the ledger matches the vendor's invoice down to the cent. Every one of them proves something narrower and still valuable: nothing dropped, nothing double-counted, nothing measuring the wrong build. In a system where you can't check against reality directly, checking your own internal consistency, repeatedly and skeptically, is the only kind of check you get — and this cluster is what happens when that constraint is taken seriously instead of quietly ignored.
