The Gate That Stayed Silent — When a Blocker Count That Drops Reads as Improvement

In the last article, I argued that the safety contract should move out of the LLM critic and into deterministic gates. The critic can vary. The gates should hold.

This one is about the failure mode I did not see coming: what happens when a gate stops working and nobody notices because the metrics look better.

The Problem That Was Hiding Inside the Solution

By v0.2.2, the blocker counts were dominated by structural families:

unsafe_sequencing: 226 blockers
unverified_dependencies: 185 blockers
weak_rollback: 86 blockers

Those numbers felt like evidence that the deterministic layer was doing its job. A high blocker count on structural categories meant the gates were firing.

Then a community reader named Artjoms Stukans pointed out the failure mode I had not modeled:

"If one blocker class stops firing after some refactor, your numbers only look better. 226 becomes 40 and that reads like plans got safer."

He described a Kubernetes incident where four releases in a row never actually ran. An old ReplicaSet kept one pod Running. Every health check passed. Every smoke test was green. Nothing said a word.

That is exactly the shape of this problem.

A deterministic gate is more reliable than an LLM critic. That is true. But it is also quieter. A non-deterministic critic advertises its own unreliability through label flips and evidence drift. A gate that silently stops firing fails the same way on every trial. Repeated runs agree. The agreement reads as confidence. You lose the variance signal exactly where the contract now lives.

Why the Existing Defenses Were Not Enough

Before Artjoms's comment, I would have said the system already had protection against this.

It did. Just not enough.

Mechanism	What it detects	Why it misses silent gate death
`underclaim_approvals` metric	LLM critic failed to block a known-bad plan	Tests the LLM critic, not the deterministic gates
Deterministic gate unit tests	Each gate fires on its fixture	CI tests pass locally; the concern is production refactor
Regression detector	New blockers appearing between revisions	Detects blockers appearing, not blockers disappearing
Drift alert	Z-score spike in severity	Tracks severity changes, not total disappearance

The gap was architectural: no mechanism asserted that each gate class still fires on a known-bad plan as a continuous health check. The gates were tested at build time. They were not monitored at runtime.

The Gate Canary

The fix is a gate canary — a deterministic, zero-LLM-cost health check that runs before every eval sweep and asserts each gate still fires on its canary bad plan.

Each gate class gets a (good_plan, bad_plan) pair:

Gate	Good plan	Bad plan
`ordering`	t1 → t2	t2 → t1
`rollback_credible`	high risk + reachable rollback	high risk + unreachable rollback
`verification_ordering`	verify before consume	consume before verified
`preconditions`	precondition references established fact	precondition references nothing

The canary runs as:

Pre-commit gate: plancritic gates canary --check — fails CI if any gate stopped firing
Eval sweep integration: canary results appear in every field-test output
Dashboard metric: one green/red per gate, visible at a glance

If a refactor or a bug silences the ordering gate, the canary turns red immediately. The blocker count dropping from 226 to 40 no longer reads as improvement. It reads as a regression that must be explained.

The Harder Lesson

The gate canary fixes the immediate gap. The harder lesson is about trust models.

I had been thinking about safety in two layers:

Layer 1: the LM critic — unstable but flexible
Layer 2: the deterministic gates — stable and authoritative

That framing is not wrong, but it is incomplete. The gates are stable only if someone watches them. Trusting a deterministic layer because it is deterministic is the same mistake as trusting a test suite because the tests pass. The question is not whether the gates are reliable by construction. The question is whether you know the moment they stop being reliable.

A deterministic safety layer does not remove the monitoring problem. It moves it.

Before the fix, the monitoring was on the critic's non-determinism (label_flip_rate, evidence_drift). After the fix, the monitoring has to be on the gates themselves — not on their outputs, but on their continued ability to fire.

That is what the gate canary does. It does not check whether the gate approved or blocked a particular plan. It checks whether the gate is still alive.

The Principle That Generalizes

If you build deterministic safety layers for LLM systems, ask yourself:

What is your equivalent of the Kubernetes ReplicaSet that kept one pod Running?

What component, if it stopped working, would produce better-looking metrics?

The answer is rarely obvious at design time. I only found mine because a reader described an incident from an unrelated system and the shape matched.

The general principle is:

Every deterministic safety gate needs a canary that fires independently of the gate's normal operation.
The canary must use a known-bad input so failure = silence is impossible to misinterpret.
The canary status must be visible in the same dashboard as the gate metrics — not in a separate CI log that nobody reads.

That is the difference between a safety layer you trust and a safety layer you have not yet watched break.

Previous PlannerCritic articles

Links