Originally published at tddbuddy.com.
Series: **Lights Out, part 2 of 3. Part 1 named the state of the pipeline the day the human gate came off. Part 3 asks what makes an unattended loop climb. Related reading: A Flaky Test Is a Corrupted Reward Signal named noise as a signal problem for agents; the same shape applies one layer up when the pipeline itself is unreliable. Tamper-Resistant Test Design Is What the Suite Now Owes the Codebase named separation of authorship as a design property; this post applies it to the merge queue.
Confidence in an unattended pipeline is an artifact, not a feeling.
A team does not develop trust in a lights-out loop by watching it, because the loop runs in the hours nobody is watching. Trust that only holds while the on-call is at their desk is not trust. It is presence. What the team actually has, after the last human gate comes off, is a list of failure modes the machinery refuses to produce. The list starts short. It grows one incident at a time. Each entry is a specific regression that the pipeline learned to reject, in a place that fails closed whether anyone is looking or not. Confidence is that list, and the discipline is adding to it faster than the pipeline invents new ways to break.
Part 1 named the state. This post pays the confidence argument through the scar this pipeline actually carries.
Confidence Is an Artifact, Not a Feeling
Two ways to think about trust in automation compete for airtime.
The first is the exposure model. Watch the pipeline long enough, see it succeed enough times, and confidence follows. That model works for a system whose failure modes are within the observation window. It fails for a system whose failures are rare, sudden, and shaped like nothing the operator has yet seen. Watching a healthy pipeline for six weeks does not tell the operator what the pipeline does on the seventh, and the seventh is where every interesting failure lives.
The second is the artifact model. Trust is the union of the failure modes the machinery has explicit defenses against. Every defense is a claim: this class of failure cannot recur silently, because the pipeline refuses to accept it. Add a defense and the trust surface grows. Remove one and it shrinks. The team's confidence is measured by the count and the coverage of those claims, not by the length of the last incident-free streak.
The artifact model is the model a lights-out pipeline can actually pay. It matches the way a test suite earns trust. The suite is trustworthy because the assertions in it refuse the behaviors the team decided were wrong. It is not trustworthy because someone is watching it run. The pipeline the team runs unattended is one layer up. Same model, same artifact, same discipline.
The 2026-05 Incident Is the Scar This Post Carries
Name the scar before defending it.
In May 2026, a post landed on main as a single line of base64. An agent-generated draft, correctly formatted at generation time, had gotten mangled by a broken serialization step somewhere between the writer and the merge. The pull request checks that existed at the time did not read the file's content structurally. They read it as text, saw it was non-empty, and passed. The bot merged the PR on its pubDate. The build broke. The site's blog index rendered a page with no readable content and a title field that had somehow parsed as valid Markdown despite being an encoded blob.
The build stayed broken for two weeks. The scheduled runs kept firing. Each one produced a new draft PR, and the queue behind the broken deploy stacked. When the pipeline was finally patched, the retry logic republished a post that had already been published, on a day it had already been published. Now there was a double-publish on the index page, a broken build in the archive, and a two-week hole in the RSS feed.
That is the receipt. No sanding. The pipeline shipped a base64 line to production, broke itself, and repeated the failure once before anyone caught it. The team's confidence in the loop, understandably, was zero on the day the fault surfaced.
The response the industry defaults to at that moment is more human review. This post argues the response was the wrong shape, and names what the right shape was.
More Review Would Have Caught It Once
Human review at a lights-out cadence is a lottery ticket, not a defense.
The 2026-05 incident would have been caught by a careful reviewer opening the PR and noticing the file was gibberish. It would have been caught once. The next agent-generated malformed post, produced by a different serialization bug six months later, would have to be caught by a reviewer who happened to look on the day it landed. The failure mode was not "the reviewer missed a specific PR." The failure mode was "no structural gate blocked a class of file." Fixing it with attention leaves the class open.
The distinction between an artifact defense and an attention defense is where the confidence argument lives. An attention defense is contingent on the reviewer being present, alert, and paying attention to the specific PR at the moment it merges. An artifact defense runs on every diff, on every commit, whether anyone is looking. Both defenses can catch the same incident once. Only the artifact defense catches its siblings.
A team with humans in the merge loop can lean on the reviewer catching a malformed post before it lands. That works when the reviewer's throughput matches the pipeline's, which is exactly the assumption Part 1 argued had already failed. A lights-out pipeline cannot lean on that reviewer. The catch has to live in the machinery, and the incident that produced the catch has to leave a specific structural change behind.
The incident is not the lesson. The pinned defense is.
Structural Gates and Pinned-Incident Tests
The 2026-05 fix took the shape it did because the incident named its own defense.
The first change was a schema gate on frontmatter. The pipeline now reads every blog post through a structural validator. Frontmatter that fails to parse, whose slug does not match its filename, whose pubDate is not a valid Monday in the future, whose homepageBullets lack a lead and body: rejected at CI time, merge blocked, no human in the loop. The gate runs on every push against every branch. It fails closed.
// astro-site/src/content.config.ts (excerpt)
const blog = defineCollection({
type: 'content',
schema: z.object({
title: z.string(),
description: z.string(),
pubDate: z.date(),
draft: z.boolean().default(false),
homepageBullets: z.array(
z.object({ lead: z.string(), body: z.string() })
).optional(),
}),
});
The second change was atomic pushes. The publish step now writes the whole post commit or none of it. There is no state where the frontmatter updated but the body did not, or where the merge landed but the deploy never rebuilt. Half-applied changes cannot exist. The 2026-05 incident had a partial state (the merge landed, the deploy started, the deploy failed, the queue kept firing) and the atomic push closes that class of failure at the pipeline layer.
The third change was pinned-incident tests. Each incident the pipeline paid the cost of learning got a named test. Reading the test file is reading the pipeline's incident log:
test('Cannot_publish_post_encoded_as_base64_blob', () => { /* ... */ });
test('Cannot_double_publish_on_the_same_day', () => { /* ... */ });
test('Cannot_publish_post_whose_pubDate_is_not_a_Monday', () => { /* ... */ });
test('Cannot_publish_post_whose_frontmatter_contains_an_em_dash', () => { /* ... */ });
Each name is a claim about a failure mode the pipeline can no longer produce silently. Each test runs on every commit. Adding a test is how the incident earns its keep. Removing one requires the same append-only rule the tests themselves earned in an earlier post: a human authors the deletion, in a separate commit, with a reason. The test file is where confidence lives. Sanding it down without saying so is where confidence quietly leaks.
Veto Windows Replace Approval Queues
The last change was structural, not code.
The pre-lights-out queue was an approval queue. Every PR waited for a reviewer to say yes, and the reviewer was the same intermittent human the failure mode of Part 1 already exposed. Removing the queue removed the fiction that the approval was happening. Replacing it with a veto window preserved the safety without preserving the bottleneck.
The window is bounded: a fixed interval between the PR opening and the auto-merge firing. During the interval, any watcher can add a hold label. A held PR is pulled out of auto-merge and lands on a human's desk with the reason attached. After the interval, the merge fires. Watchers are not required to be present. They are required to be reachable, which is a much cheaper property to guarantee.
The veto window is the tamper-resistant test rule applied to the merge queue. The default direction of the machinery favors the throughput; the human is the exception path. The reviewer is not proving the PR is safe. The reviewer is proving the PR is unsafe when they think it is. Their absence is not a block. Their intervention is.
Standard regression prevention is what the discipline looks like. The novelty is not the practice. The novelty is that every catch has to live in the machinery, because there is no reviewer in the merge loop to lean on. That constraint is what turned the 2026-05 incident into a defensive posture instead of a story.
Confidence Is Cumulative and Attributable
Return to the opening claim. Confidence in an unattended pipeline is an artifact, not a feeling.
The 2026-05 scar produced a schema gate, an atomic push, four pinned tests, and a veto window. Each is a receipt for a failure mode the pipeline can no longer produce silently. Each runs whether the team is watching or not. Together they form the trust surface a lights-out loop actually rests on, and the surface is measurable: count the gates, count the tests, count the invariants the machinery refuses to violate. That count is the answer to "how much do you trust this."
The list is not complete. It will never be complete. New failure modes will surface, and the discipline is to name each one and pin it as it arrives. Confidence is cumulative: every named defense outlasts the incident that produced it. Confidence is attributable: every defense points at a specific commit, a specific scar, a specific class of failure. That attribution is what distinguishes earned confidence from the exposure-model version, which points at nothing and evaporates the first time the pipeline surprises anyone.
A human is still in this system. The human is in the veto loop, in the editorial layer, in the queue that decides what gets written at all. The human is not in the merge queue anymore, because the merge queue's job moved into the machinery on the day the 2026-05 scar earned its keep.
Part 3 asks whether a loop like this can climb. This post ends at the answer to a different question: whether a loop like this can be trusted, unattended, at all. It can, and the shape of the trust is the artifact list. Watch nothing. Read the list. If the list is short, the confidence is short. If the list grows faster than the pipeline invents new failures, the trust surface grows too.
The scar came first. The list came second. The list is what the team runs on now.
