I Built a Frozen-Scan Detector. A Week Later It Fires Daily — and Nothing Consumes It

This is the verdict my detector printed ten minutes ago, verbatim:

$ python tools/scanfresh/scan_freshness_guard.py check \
    --ledger recommendations/signal_history.jsonl \
    --verify recommendations/signal_verify_log.jsonl \
    --panel-dir history_ext --today 2026-10-19

verdict: FROZEN
  ! panel newest bar 2026-09-28 is 15 business days behind 2026-10-19
  ! trailing identical run of 15 entries (2026-09-28 -> 2026-10-16, fp=4c29e76725)
  ! 2 verification transition(s) advanced target_date without changing any measured detail

Exit code 2.

Three numbers:

• 15 consecutive sessions whose ledger entries are identical after stripping every timestamp.
• 15 business days how far the data panel those sessions claim to analyse has drifted behind today.
• 3 — how many of those identical entries were written after I shipped this detector.

The third number is the one this post is actually about.

The pipeline

A local Python job over a panel of per-stock daily bars:

panel (3,147 JSON files, one per stock)
  -> classify every stock into one of six technical patterns (build_dataset_v3.compute_tech)
  -> append {scan_date, dist, picks} to recommendations/signal_history.jsonl
  -> next session, score yesterday's picks, append to signal_verify_log.jsonl

It is a pure function of the panel. Which is the entire bug surface: if the panel stops being updated, the scan keeps succeeding at analysing the past.

What "success" was measuring

The job's contract was "append a line and exit 0". It honoured that contract 21 times in a row. Every log line green, every night at 15:35, Weekdays only.

A verification ledger is supposed to answer "were yesterday's picks right?". Mine answered with the identical four per-stock returns under three different target dates, because it was re-scoring a price file whose mtime had not moved since 2026-09-28 12:10.

The panel detail that makes this concrete: 3,147 files exist, but only 3,049 instruments appear on the newest bar. The other ~100 files stopped at earlier dates. Nothing complained — a file count is not a freshness measurement.

Fingerprint the answer, not the activity

Activity metrics — ran / exit code / rows processed / wall time — are all satisfiable by a frozen pipeline. The one reading that a frozen pipeline cannot fake is did the substance of the output change?

So hash the fields that define the answer, and deliberately exclude the ones that define the run:

SNAP_KEYS = ('dist', 'picks', 'rebound', 'breaks')

def fp(obj, keys=SNAP_KEYS):
    picked = {k: obj[k] for k in keys if k in obj}
    if not picked:                      # unknown shape -> whole record minus timing
        picked = {k: v for k, v in sorted(obj.items()) if k not in
                  ('scan_date', 'verify_date', 'target_date', 'generated', 'run_at', 'ts')}
    blob = json.dumps(picked, sort_keys=True, ensure_ascii=False).encode()
    return hashlib.sha256(blob).hexdigest()[:10]

sort_keys=True so re-serialising a dict cannot fake a change. scan_date excluded so a date bump cannot fake one either. Anyone can recompute these fingerprints from the ledger with no access to my code.

Three checks fall out of that:

Two of the three also catch a subtler failure I had not thought about until I wrote the check: a pipeline can keep reading while its numbers stop moving, and it can keep re-dating while its measurements stay frozen. Both look like output growth. Neither is.

One week later: the detector is right, and irrelevant

Here is the honest part. I built this, published the tool, wrote about the failure mode — and then never wired it to the writer. Since the guard landed, the writer cron has appended three more byte-identical entries. The guard's verdict has been 2 every day. Nothing reads exit codes of a tool nothing calls.

So the real lesson of this week is not "add a staleness check". It is:

A detector whose verdict is not a precondition is a dashboard.

A green dashboard on a frozen pipeline is worse than no dashboard, because now there are two artifacts telling you everything is fine: the writer's exit 0 and the detector's existence.

The fix is one line in the writer, and I am writing it here so I have to do it:

if subprocess.run([sys.executable, 'tools/scanfresh/scan_freshness_guard.py',
                   'check', '--ledger', LEDGER, '--panel-dir', PANEL,
                   '--today', today]).returncode == 2:
    sys.exit('refusing to append a snapshot I cannot vouch for')

The snapshot must not be written at all. Logging the alarm while writing the stale entry anyway is the same bug with extra ceremony.

Rules I now apply to any scheduled analysis job

1. Green is not evidence. "Ran, exit 0" is a statement about the runner. Ask which reading would have been different had the job failed. If the answer is "none", you measured nothing.
2. Ask what the job's output would look like if it were secretly frozen. That question is cheap to answer and it is where all my reliability bugs live.
3. Fingerprint the answer, exclude the clock. Timestamps, run ids, row counts and file counts all change when nothing has changed — and none of them change when everything has.
4. "No input" needs its own exit code. The guard returns 3 INSUFFICIENT when handed nothing to judge, because a checker that examined zero entries must never print "healthy" — the same reason a file scanner that opened zero files must never print "clean".
5. Two identical days are normal; fifteen are not. Make the threshold explicit, tune it against your own hit rate, and remember that a single repeat is not a signal.
6. Gate, don't report. A verdict nobody consumes is decoration. The check belongs in the writer's precondition, and the failure mode should be no snapshot written, not snapshot written plus a warning logged.

The detector

tools/scanfresh/scan_freshness_guard.py — stdlib only, 409 lines, exit codes 0 HEALTHY / 2 FROZEN / 3 INSUFFICIENT / 1 error, with an 18-assertion selftest over synthetic fixtures:

python tools/scanfresh/scan_freshness_guard.py selftest          # 18/18 passed

python tools/scanfresh/scan_freshness_guard.py check \
  --ledger  samples/real_frozen_history.jsonl \
  --verify  samples/real_verify_frozen.jsonl \
  --today 2026-10-12                                             # exit 2, FROZEN

samples/ ships the real frozen ledgers (the 16-entry one with the trailing run of 10, and its verification ledger), so the FROZEN verdict is reproducible on your machine without my data, plus a synthetic healthy ledger so the pass path is testable too.

Code, samples, notes: https://github.com/Felixwang007/awesome-content-tools/tree/main/tools/scanfresh — MIT, take the 400 lines and skip the essay.

One caveat I would rather state than have you discover: this detects frozen output, not a broken fetch. A source that returns different garbage every day passes all three checks. Freshness is necessary, not sufficient. The obvious next gap in mine is a China exchange holiday calendar — right now panel_stale knows weekends but treats October's exchange holidays as ordinary business days, which is exactly how a 15-day gap reads as ordinary Monday drift.

If you build analysis agents you actually run daily — screening, scheduled scans, anything that appends a snapshot and exits 0 — the same guard is packaged as a skill on 虾评 (xiaping.coze.com) next to the A-share three-pillar screener, and the raw script is in the repo above.

All numbers here are from a single check run against the real ledger and panel on 2026-10-19; the ledger, verification log and panel are a private workspace, but the guard and the exact frozen ledgers it trips on are public in samples/.