What to alert on when your LLM feature is live

Your LLM passed the vibe check. It is in production. Nothing is throwing exceptions, and users are still clicking. Silent failures can develop while everything appears healthy. Charity Majors argues that the source of truth is what the system is doing right now. Malte Ubl’s path from quick validation to shipped agents adds the other half: once business logic lives in prompts and tool loops, you need traces before you tune prompts or models.

If you only watch HTTP 500s, you may learn about model drift from support tickets. Drift is performance getting worse while the weights file stays the same: inputs shift, behavior shifts, errors do not. Pair that with instruction adherence, because a fluent answer that ignores your format still breaks downstream code. Alerts should treat obedience and quality as separate signals.

For a fuller map of how observability, evaluation, and infrastructure fit together, the Topics index on Chain of Thought is a useful anchor. This article is narrower: what to page on, and how each alert links back to one trace.

Why does production need different alerts than pre-launch evals?

Pre-launch checks answer “does this work on yesterday’s examples?” Production answers “what is happening on this request, with this user, with this tool output?” Majors describes teams that debug by finding whoever last changed the code and asking the resident expert. Generated or agent-written paths break that habit: there may be no expert, and the failure is nondeterministic.

Like, the source of truth is what is happening right now.

Charity Majors, Honeycomb, on Chain of Thought ep 18

Malte Ubl frames the same shift for agents: core logic is partly prose, execution paths vary, so logging alone is not enough. You alert on aggregates, but you investigate on traces.

Decision rule: if the feature runs more than once per user session, or calls tools, default to trace-backed alerts. If it is a single-shot completion with a strict schema, lean on validators plus sampled trace review.

What should page me on drift versus instruction adherence?

Drift alerts watch live distributions and outcomes rather than one offline score. The glossary's drift definition says to watch live inputs and outputs. These are the triggers I would propose:

• Input side: new clusters of prompt length, language, or entity types versus a rolling baseline (weekly is a fine start for many apps).
• Output side: rising rate of human “wrong” feedback, thumbs-down, or support tags that mention the feature.
• Model side: if you route across models or versions, alert when one variant’s error or rewrite rate diverges from siblings.

Instruction adherence alerts are mechanical where you can be mechanical. Schema validation failures, missing required JSON fields, over-length responses, and regex checks on forbidden patterns should increment a counter and page when the rate crosses a threshold you set from staging traffic. For fuzzy rules (“formal tone”, “cite the doc”), use an LLM-as-judge on a sample, but page on validator failures first because those are cheap and unambiguous.

Do not merge these into one “quality score” alert. A model can stay on-distribution while ignoring instructions, or follow format while becoming less helpful.

How do latency and cost fit the same playbook?

Ubl points out that extra LLM calls add latency and cost. I would make p95 latency and dollars per successful task first-class SLOs for agents and multi-step chains. Alert when:

• Latency: p95 or p99 wall time per trace exceeds budget, or tool-call count per trace spikes (often a loop or retry storm).
• Cost: tokens in plus tokens out per trace, or per successful business outcome (ticket resolved, row written, PR opened), jumps relative to a seven-day baseline.

Tie both metrics to the same trace id you use for quality. My working hypotheses: a cost spike with flat quality may point to a prompt change or a model swap, and a latency spike with rising validator failures may mean the model is rambling before it breaks schema.

How do I connect an alert to a trace I can act on?

An alert without a trace is a mood. Minimum fields per LLM span (sketch, not a vendor schema):

# Sketch: one alert should resolve to one trace_id
def on_alert(alert_name: str, trace_id: str, store):
    trace = store.get_trace(trace_id)  # turns, tool I/O, model id, tokens
    if alert_name == "schema_fail_rate":
        return [s for s in trace.spans if s.get("validator") == "json_schema"]
    if alert_name == "cost_baseline":
        return trace.summary(tokens_in=..., tokens_out=..., model=...)

Workflow when a page fires:

1. Open the exemplar trace attached to the alert (not an average).
2. Read tool returns before blaming the model (Ubl flags tool responses as a prompt-injection entry point).
3. Compare model id, prompt version, and tool definitions on that trace to the last known good deploy.
4. If the failure is adherence, fix validators or tighten tool docs before you swap models.
5. If the failure looks like drift, refresh eval cases from recent production samples and re-run before you retrain or fine-tune.

Ubl’s production path assumes this loop exists before optimization:

you need tracing now. Because the trace tells you what the thing did.

Malte Ubl, Vercel, on Chain of Thought ep 39

He also notes traces are not identical run to run, which is why the alert should carry a bad example along with the metric delta.

What belongs in the first week after launch?

Start small and explicit:

Run a weekly review: one drift-oriented alert, one adherence alert, one latency or cost alert. If you cannot name the trace id you would open for each, the alert is not ready.

Checklist

• Separate alerts for distribution or quality drift and for instruction adherence failures (validators first, judges second).
• Page on p95 latency and cost per trace as well as HTTP errors.
• Attach at least one exemplar trace id to every alert payload.
• When investigating, read tool outputs on the trace before changing models.
• Refresh offline evals from production traces when drift alerts fire, rather than only tweaking prompts in the dark.

More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.