The first three parts covered thesis, cost, and security. This one covers everything that separates a demo from a production system. None of it is exciting. All of it is why some teams sleep at night.
The production checklist
Trace everything. Langfuse, LangSmith, or OpenTelemetry; pick one and stick with it. When (not if) a workflow misbehaves in production, traces are the difference between debugging and staring into a black box.
Test like it's software, because it is. Unit tests, integration tests, CI. Accept a hard truth too: you will never reach full coverage of model behavior. That's why critical paths get verification loops and everything else gets user feedback loops.
LLMs are not calculators. Force them through tools for arithmetic, with strict JSON schemas on inputs and outputs. Validate at every boundary.
Design for failure everywhere:
- Network requests fail, so retry with backoff
- Servers go down, so replicate
- Agents loop, so put step limits in the harness
- Humans and LLMs wreck databases, so keep immutable offsite backups
Make workflows resumable. Durable execution engines and checkpointing mean a crashed 40-step run resumes at step 31 instead of restarting. Every tool call should be idempotent too, so retries don't double-charge anyone.
Run agent step ──► failed? ──yes──► resume from checkpoint ──► retry idempotent tool
│ │
no └──► retries are safe because every call is idempotent
▼
Next step ──► log + trace + meter cost
Roll out like an SRE, not a shipper. Controlled rollouts, canaries, versioned prompts, feature flags, and every change revertible in one action. Model updates will silently shift behavior. You want to be the one who notices through monitoring and alerts, before your customers do.
Meter money obsessively. Cost limits at every level: per chat, per user, per tenant, per SaaS account. One buggy retry loop without budgets is how cloud bills become front-page news internally.
Keep models swappable. Provider outages and price hikes are routine. An abstraction layer over your LLM calls turns those from incidents into config changes.
One thing to stop doing: treating the harness as an afterthought. The harness (schemas, limits, retries, locks, secrets) is the product. The model is a component inside it.
Your action
Audit your current setup against this list and find the missing row. No tracing? No cost budget per tenant? No revert plan for prompt changes? Fix the scariest gap this week. Each of these costs less than the incident it prevents.
Final part next: Part 5, Humans in the Loop Without Burning Out Humans. Because someone still has to click approve.
