My LLM Critic Flip-Flops on Every Run. That's Fine — Because a Frozenset Decides What's Fatal.

This is a companion to the PlannerCritic series. Article 2 was about a specific critic bug. This one is about the design principle I extracted from fixing it — and the measurement that proved it holds.

I measured my LLM critic on identical input across five trials. It returned a different verdict every single time. label_flip_rate = 1.0. It also never let a defective plan through. underclaim_approvals = 0. Both are true. The frozenset is why.

The Uncomfortable Measurement

In v0.2.1 I added a test I'd been avoiding: send the same boundary-case plans through the real critic model five times and measure what changes. The live-critic boundary evaluator (#218).

The numbers:

Read those first two rows carefully. On the same plan, five times in a row, the critic returned a different verdict and a different reasoning every time. If you were betting the safety contract on the critic being consistent, you'd have lost.

I wasn't betting on that. But it was still uncomfortable to see it measured at 1.0. "Mostly consistent" would have felt safer than "maximally inconsistent." Maximally inconsistent is what we got.

Why the Safety Contract Doesn't Care

The critic is 100% non-deterministic on verdict. It is 0% under-claiming on seeded defects. Both numbers are real. The reason both can be true at once is that they measure different directions, and the architecture assigns them to different owners.

There are two ways an LLM critic can be wrong:

1. Under-claim — a defective plan gets zero blockers. This is the dangerous direction: a bad plan slips through.
2. Over-claim — a sound plan gets a blocker for "not thorough enough." This is the noisy direction: a good plan gets escalated.

The architecture gives each direction to a different authority, and neither authority is the LLM.

The deterministic gates own the under-claim direction. Preconditions, topological ordering, rollback credibility, verification ordering — these parse the plan's AST, not its prose. They cannot be prompt-injected because they don't read natural language (Article 5 covers this). A defective plan that the LLM critic happens to miss on trial 3 still gets caught by the gate that checks whether every precondition is established by an earlier task. That's why underclaim_approvals = 0 despite label_flip_rate = 1.0.

A code-enforced allowlist owns the over-claim direction. This is the part I want to dwell on, because it's the part I learned the hard way.

The Frozenset Is the Contract

In Article 2 I told the critic to be "an adversarial plan reviewer." It obeyed. It blocked plans for being incomplete — "this plan could also cover edge case X" — not for being unsafe. Every strict goal escalated for the wrong reason.

The fix wasn't more prompt engineering. I tried that first; the critic still escalated completeness concerns to blocker about 30% of the time. The fix was a frozenset:

_BLOCKER_ELIGIBLE_FAMILIES = frozenset({
    "unsafe_sequencing",
    "weak_rollback",
    "unverified_dependencies",
    "feasibility",
})

if severity == Severity.BLOCKER and item.heuristic_family not in _BLOCKER_ELIGIBLE_FAMILIES:
    severity = Severity.WARNING

Even if the LLM returns blocker for a risk or missing_steps finding, the code downgrades it before it enters the findings list. After the fix, zero advisory findings appeared as blockers across 92 post-fix runs.

The prompt is helpful. The frozenset is the contract.

That's the sentence I kept coming back to. The LLM is allowed to be wrong about severity — and it is, on every trial — because the code doesn't trust the LLM's severity label. It trusts the structural property (which family the finding is in), which the gate derived deterministically. The LLM's label is decorative; the family is load-bearing.

The General Principle

Here's the part I think generalizes beyond my project.

When you put LLM judgment on a critical path, you inherit every vulnerability of LLM judgment — non-determinism, prompt sensitivity, the tendency to be "thorough" when you asked it to be "adversarial." When you keep the critical path deterministic, you get resistance by design — but only to the things your code can check structurally.

The split that worked:

• Code is authoritative everywhere it can be. Ordering, preconditions, rollback presence, schema. These have a right answer the AST can verify.
• The LLM is advisory everywhere it could be wrong. Severity calibration, completeness, "is this rollback credible." These are judgment calls where the LLM adds value but can't be trusted to be consistent.

The mistake is putting the LLM in charge of the second category and making its verdict load-bearing. The fix is letting the LLM inform the second category while code decides whether its verdict counts. The frozenset is the mechanism: the LLM proposes a severity; code checks whether the structural property supports it; code wins ties.

You can see the same pattern in how approval works. In approval.py, a gate blocker is always a hard blocker; an LLM critic blocker is probabilistic and posture-dependent. The gate vetoes are deterministic. The critic's contribution is downgraded to advisory under postures that can't afford false escalations.

The Honest Seam

I want to be careful not to oversell this. The deterministic authority has a real limit, and it's the one a commenter (@ethanwritesai) sharpened for me: the frozenset only works because the family a finding belongs to is itself derived deterministically. If the LLM could mislabel which family its finding is in, the allowlist would be trusting a label again. Right now the family comes from the gate or the critic's structured output, and the critic's structured output is the soft edge.

That's tracked now — the property-vs-label distinction is part of the v0.3.0 critic-satisfaction work (#254). The principle holds; the implementation has a seam where the LLM's self-classification still leaks in. I'd rather name it than hide it.

There's a second, more obvious seam: the deterministic gates only check structure, so a well-formed malicious plan — dummy rollback, dummy verification — satisfies the linter. Deterministic authority is necessary; it is not sufficient. That seam (and the indirect-injection surface that feeds it) is the subject of its own piece — see I Published Every Flaw My Safety Tool Can't Catch. I'm keeping this article about the frozenset and that one about the holes.

What I Stopped Doing

I stopped trying to make the critic consistent. I spent a while in v0.1.0 tuning the prompt to get stable verdicts. It didn't work, and the boundary evaluator finally told me why in numbers: the ceiling is 1.0, not because the prompt is bad but because the model is non-deterministic by construction. You don't fix that with prompt engineering. You route around it by not depending on it.

I also stopped treating "the critic flagged something" as a reason to escalate. Under balanced posture, a critic blocker is a warning, not a veto — because the critic is allowed to be wrong in the over-claim direction, and the cost of a false escalation is a human's time. The deterministic gates are the veto layer. The critic is the "you should probably look at this" layer.

And I stopped trusting the LLM to decide what was fatal. That's the whole title of this piece. The LLM is allowed an opinion about fatality. The frozenset decides.

Questions I'm Still Sitting With

• Where's the LLM on your critical path? Not where you think it is — where it actually is, in the code that runs when a decision matters. I was surprised by my own answer when I traced it.
• What's the one invariant you'd hand to code instead of the model? For me it was severity. For you it might be tool-selection, or when to escalate, or whether a response is "done." The test is: does it have a structural property code can check, or is it genuinely a judgment call?
• Does "the LLM is advisory" actually hold in your system, or does "advisory" quietly become "authoritative" when no one's watching? This is the one I check on myself now. The frozenset is only the contract if nothing downstream reads the raw severity label.

I don't think deterministic-first is the whole answer. I think it's the floor. The interesting work is what you build on top of the floor — and how honest you are about where the floor ends.

Series: Article 1 · Article 2 · Article 3 · Article 4 · Article 5

Links:

• Repo: github.com/deghosal-2026/planner-critic-engine
• The severity guardrail (Article 2's fix): critique/critic.py
• The approval contract: approval.py
• The boundary evaluator that produced the 1.0 numbers: eval/live_boundary.py
• v0.3.0 follow-ups: #249 (indirect injection) · #254 (critic satisfaction / property-vs-label)