A crashed service pages someone at 3am. A model that's quietly gotten worse ships a clean 200 and nobody finds out for six weeks.
That asymmetry is the actual bug in most production AI. Every other component in your stack has a failure signal you can alert on — a non-zero exit, a timeout, a stack trace, a p99 that walks off the chart. A classifier that's drifted has none of those. It returns the same shape of JSON it returned yesterday, at the same latency, with confidence scores that look fine, and the content inside is wrong. Your monitoring is green. Your uptime is 100%. Your approval queue is full of garbage.
The numbers around this are grim and specific. In 2025, 42% of companies abandoned most of their AI initiatives, up from 17% the year before, and large enterprises killed an average of 2.3 projects each at roughly $7.2M in sunk cost per initiative. What gets reported as "the AI didn't work" is usually something narrower: it worked, it stopped working, and the gap between those two events went unobserved long enough that trust never came back.
So instrument the gap. Here are three guardrails, in the order I'd add them to a system that has none.
1. A golden set that runs like a unit test
The cheapest correctness signal you can build is a frozen set of hand-labeled cases and an assertion. Not an eval harness. Not a dashboard. A test file.
# tests/test_golden_set.py
import json
from approvals import classify
GOLDEN = json.load(open("tests/golden_set.v4.json")) # 200 cases, hand-labeled
def test_golden_set_accuracy():
wrong = [c for c in GOLDEN if classify(c["input"]) != c["label"]]
accuracy = 1 - len(wrong) / len(GOLDEN)
assert accuracy >= 0.92, (
f"accuracy {accuracy:.3f} below floor; "
f"regressed: {[c['id'] for c in wrong][:5]}"
)
The part people skip: run it on a schedule, not just on pull requests. Your code didn't change. That's the whole point. A schedule: trigger in GitHub Actions, every night, on the production model and the production prompt:
on:
schedule:
- cron: "0 6 * * *"
pull_request:
Two traps worth naming. First, golden sets rot — the distribution they represent is the one you had when you wrote them. Version the file (v4, not golden_set.json), append ten fresh production cases a month, and never quietly delete a case that starts failing. That deletion is the drift, recorded. Second, don't assert on a single accuracy number if your classes are lopsided. A 94% accurate spam filter that has stopped catching spam entirely still scores 94% when 94% of traffic is legitimate. Assert per-class recall.
2. Watch the inputs, because they move first
Golden sets tell you the model got worse. They can't tell you it's about to. Labels arrive late — sometimes weeks late, sometimes never — but the inputs arrive in real time, and they shift before your metrics do.
Population Stability Index is the boring, effective tool here:
import numpy as np
def psi(baseline, current, bins=10):
edges = np.quantile(baseline, np.linspace(0, 1, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
b = np.histogram(baseline, edges)[0] / len(baseline)
c = np.histogram(current, edges)[0] / len(current)
b, c = np.clip(b, 1e-6, None), np.clip(c, 1e-6, None)
return float(np.sum((c - b) * np.log(c / b)))
Rough reading: under 0.1 is stable, 0.1 to 0.25 is worth a look, above 0.25 means the thing you're scoring today isn't the thing you built for.
Text inputs don't have a natural histogram, so project them onto a scalar first and PSI that. Cosine distance from last quarter's embedding centroid. Token length. Share of requests matching your top-20 known intents. Fraction containing a product name you shipped after the prompt was written — that last one caught a real regression for me, because the model had never seen the feature it was being asked to route.
Log the number daily. A drift metric you compute once during an incident is archaeology, not monitoring.
3. Fail closed, and put a hard ceiling on the spend
The first two guardrails are detection. This one is containment, and it's the one that decides whether a bad Tuesday is an incident or a shutdown.
Give the system an explicit third answer. Not approve, not reject — abstain, and route to a human:
FALLBACK = "human_review"
def decide(case, drift_score):
label, confidence = classify_with_confidence(case)
if drift_score > 0.25 or confidence < 0.80:
metrics.incr("approvals.fallback")
return FALLBACK
return label
Now alert on the fallback rate, not on correctness. "The model is punting 3x more than it did last month" is a signal that exists today, requires no labels, and arrives weeks before anyone files a complaint.
The cost side matters more than people expect, because runaway spend is often drift's first visible symptom. When inputs stop matching the prompt, agents retry, re-plan, and re-query — the loop that cost you $40 a day starts costing $400, and it looks like traffic growth until you read the traces. Budget alerts fire after the money's gone; rate limits cap requests per second, not dollars. What you want is pre-flight enforcement: refuse the call before it reaches the provider. baar-core does exactly that — an open-source Python library that raises a 402 on a call that would exceed the cap, with atomic reservation so twenty parallel workers each "under budget" can't jointly blow through it (pip install baar-core). For teams that need the same enforcement per-user with a dashboard on top, that's noburn.dev — it blocks the API call before it fires when a user crosses their budget, rather than emailing you about it afterward.
The part nobody puts in the budget
All three of these are maybe two days of work. They almost never get built, and the reason isn't difficulty.
Shipping the model is a project, with a sponsor, a launch date, and someone's quarterly goal attached. Keeping it correct is an on-call rotation nobody staffed. There's no demo, no launch post, no line item. So the golden set doesn't get written, the drift metric doesn't get logged, and the system runs unobserved until it's wrong loudly enough that killing it is the obvious call — at which point the sunk cost makes a rebuild politically impossible.
Write the tests during the build, while the budget's still open. Retrofitting observability onto a model that's already lost the room is a much harder conversation than adding a cron job in week two.
What's the longest one of your models ran wrong in production before anyone noticed?
Originally published at https://www.robatdasorvi.com/chapters/ai-agents/why-most-ai-automation-dies-within-six-months-of-going-live
