My Canary Reviewer Trusted Its Own Memory. I Made Every Verdict Carry a Receipt.

Every deployment reviewer has a blind spot, and mine followed me around for two days: it remembered the system the way it looked before the deploy. I spent two days asking a free model to review a canary release, and it nearly approved a broken version because the baseline still looked normal in its context. How do you trust a reviewer that cannot tell yesterday from now?

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The setup that looked safe

I used MonkeyCode's free model access and its free server option for a deliberately boring batch job: a cron job that fetches error rate and p95 latency every five minutes, packs the last half hour into a JSON envelope, and asks the model for a canary verdict. A plain Python script writes each verdict to a file so I can inspect the reasoning trail afterward. There was no streaming, no real-time control loop, only a reviewer that watched the release and told me when to worry.

import hashlib, json, time

def build_envelope(metric: str, points: list[dict]) -> dict:
    payload = {
        "metric": metric,
        "window": "last_30_minutes",
        "observed_at": int(time.time()),
        "points": points[-6:],  # the six newest samples, nothing older
    }
    raw = json.dumps(payload, sort_keys=True).encode()
    return {
        "envelope": payload,
        "checksum": hashlib.sha256(raw).hexdigest()[:12],
    }

The trap was hiding in plain sight: the prompt said now, but the model has no clock to verify that claim. Now is not a value you can pass into a prompt; it is a relationship between an event and a reader. I learned that relationship is exactly what a canary verdict depends on.

Day one: the canary changed, the reviewer did not

About eleven hours into the run, the canary error rate jumped from 0.2% to 1.8%, and the model answered go with a calm note: error rate stable at 0.2%, no action required. The catch was that 0.2% belonged to the pre-deploy baseline that had been sitting in the conversation history since the first cron tick. The model did not hallucinate exactly; it trusted the most confident-looking number, and that number was a memory rather than a measurement.

{"verdict":"go","reason":"error rate stable at 0.2%","evidence":[]}

My gate logged that verdict and, because evidence was empty, turned it into a needs-attention page. The canary survived, but only because I had accidentally written a fail-closed parser. No evidence array, no timestamp, no way for any script to know which 0.2% the model meant; the repair was embarrassingly small: demand a receipt with the verdict.

The receipt rule

From that moment on, the envelope carried a checksum, and the prompt required the model to echo it back together with one evidence entry per metric containing a real as_of timestamp. The verdict automatically becomes untrustworthy when the receipt is missing, mismatched, or older than five minutes. Deterministic code, not a second model, owns that judgment.

import json, time

def verify_receipt(raw: str, expected_checksum: str, max_age: int = 300) -> bool:
    out = json.loads(raw)
    if out.get("checksum") != expected_checksum:
        return False
    for evidence in out.get("evidence", []):
        if time.time() - evidence["as_of"] > max_age:
            return False
    return True

The prompt that finally worked reads like a contract rather than a suggestion:

SYSTEM = """
You review a canary deployment.
The envelope below is the only current data.
Values labeled "baseline" are pre-deploy references, never current observations.
Return strict JSON only:
{"verdict": "go|no_go|needs_attention",
 "checksum": "...",
 "evidence": [{"metric": "...", "as_of": 0, "value": 0.0}]}
Each evidence entry must copy observed_at from the envelope. Never invent a timestamp.
"""

Asking one model to review another model is just recursive memory trust, and I had already seen what that produced on day one. The receiver of the verdict now guards the floor instead of the writer.

Day two: the ways it still broke

The model started inventing timestamps that looked plausible but pointed at its own generation time instead of the metric sample time; the freshness gate caught each one. The lesson there was to reject on the first stale piece of evidence rather than averaging, because averaging rewards confident time travelers. The second failure appeared when the model wrapped the JSON in markdown and dropped two fields, which made my parser fall back to an empty evidence list. Empty evidence now counts as a rejected verdict, because a reviewer without receipts is a rumor with a confirm button.

The third failure was my own over-correction: I labeled the baseline so strongly that the model started calling every stable pre-deploy number a danger, which produced false no_go verdicts for four straight cycles. The fix was to keep the baseline in the envelope but tag it with before_deploy and never let the model compare values on its own; comparisons belong in the deterministic gate.

The decision table

The change that made this table useful was moving all time arithmetic and all comparisons out of the prompt. The model describes what it sees; the gate decides which version of the present is real.

What I would do again

• Keep the envelope tiny: metric name, six points, observed_at, checksum, nothing else.
• Make the model copy values from the envelope instead of reasoning from vague memory.
• Fail closed on missing evidence; a missing timestamp is a rejected verdict, not a maybe.
• Keep the baseline inside the envelope but label it, and never let the model compare numbers across labels.

Limitations and who should skip this

The pattern only works when your metrics can fill a five-minute window with meaning; a low-traffic side project will starve the envelope and produce noisy verdicts. If your system needs sub-minute rollback decisions, do not route them through an LLM at all; deterministic alerting should own that loop for exactly this reason. The free server option handled a five-minute batch job without drama, but this setup is offline review, not a control loop. If you build a free-model review loop, start with the receipt rule and see how many of your assumptions are stale; mine took eleven hours to reveal the first one.