Every AI demo I've seen works beautifully. Five boxes on a whiteboard, one API call, an answer in under a second, and everyone in the room nods.
Then it meets real users. The model provider has a bad afternoon. Retries pile up into a retry storm. An agent issues the same refund twice. The RAG assistant cites a policy that was replaced two years ago. Someone pastes a document that tells the bot to email out the customer list. And at the end of the month, a token bill arrives that nobody budgeted for.
None of these are model problems. They're system design problems, and most of them have solutions backend engineers already know: timeouts, idempotency keys, least privilege, caching, tracing, budgets. They just need to be applied to a component that is slow, expensive, non-deterministic and occasionally wrong with total confidence.
So I made System Design for the AI Era, an 11-part animated series that walks through each of these problems and the architecture that handles it. It's a little over two hours in total, free, and focused on trade-offs rather than any specific vendor or framework.
Full playlist: https://www.youtube.com/playlist?list=PLOooQo0oCPgU
Here's what each episode covers and the one idea I'd want you to take away from it.
1. From demo to production architecture
We take a five-box prototype and turn it into a production architecture: a gateway, an orchestration layer, a model adapter with timeouts, bounded retries and circuit breakers, guarded workflows for consequential actions, rate limits, queues and budgets, grounding and validation, and end-to-end observability.
Takeaway: the model proposes, your application decides. Anything with real-world consequences (refunds, emails, deletes) goes through code you control, not through the model's say-so.
2. Model routing
Sending every request to your biggest model is like paging a senior surgeon to put on a bandage. This episode designs a model router: where it sits, the ladder of strategies (static rules, feature rules, classifiers, cascades), and how fallbacks have to respect compliance even on a bad day.
Takeaway: hard constraints (data residency, compliance, capability) come before cost. And evals, not vibes, decide which model is good enough for which task.
3. Putting AI agents on a leash
Your assistant graduated to an agent, with access to email, a CRM, refunds and, for some reason, a shell. We cover loop limits and stop conditions, least-privilege tools with scoped credentials, risk-tiered human approvals that don't turn into rubber stamps, indirect prompt injection, and a kill switch.
Takeaway: an agent should act with the permissions of the user it's serving, never with a god-mode service account.
4. RAG that doesn't lie
The assistant cited its sources. The sources were the problem: an old draft, a confidential spreadsheet, a policy split across two chunks. This episode covers structure-aware chunking, hybrid search with reranking, versioned and fresh indexes, permission filtering inside retrieval, and claim checks.
Takeaway: evaluate retrieval separately from generation. If the right document never reaches the model, no prompt will save you.
5. Evals that aren't vibes
Four examples and a good feeling is not a test. We build an evaluation pipeline: behaviour-based eval sets sliced by risk, code checks plus calibrated model judges, evals as regression gates in CI, shadow traffic and canary rollouts.
Takeaway: every production failure should become tomorrow's test case.
6. Cost and latency
AI summaries launched, users loved them, then the invoice arrived. This episode is about metering per feature and per outcome, trimming prompts and capping output, routing to the smallest good model, caching safely, streaming and deadlines, and moving work to batch lanes.
Takeaway: your AI bill is a design decision. Most of the savings come from sending less and reusing more, not from negotiating a cheaper model.
7. Observability
The logs said 200 OK. The customer said the bot promised a refund that didn't exist. We cover traces that record model, prompt and index versions, careful content logging (redacted, sampled, short retention), quality scoring in production, and debugging one bad answer from trace to fix to eval case.
Takeaway: a successful HTTP status tells you nothing about whether the answer was right. Measure quality, not just uptime.
8. Security and prompt injection
A document told the assistant to email out the customer list, and it tried. This episode covers threat modelling, direct and indirect prompt injection, data leakage and tenant isolation, unsafe output handling, excessive agency and cost attacks.
Takeaway: your system prompt is not a firewall. Enforce security outside the model, where text can't argue with it.
9. Memory
Forty messages in, the assistant asked for the order number it was given in message one. We design memory properly: the context window as a budget, summaries and pinned state, long-term memory with sources and expiry, per-user isolation, and keeping users in control of what's remembered.
Takeaway: a context window is not memory. Memory is data, and it needs the same care as any other user data.
10. Durable and multi-agent workflows
The agent was on step 31 of 40 when the server restarted. It started over, and every customer got the welcome email twice. This episode covers durable execution with checkpoints, exactly-once side effects with idempotency keys and compensation, pausing for human approval, and when multiple agents help versus hurt.
Takeaway: long-running AI work is a workflow problem. Treat each step like a job that might be retried, because it will be.
11. The LLM gateway
Seven teams, seven provider keys, four retry libraries, and nobody knows the total bill. The finale designs an LLM gateway and platform: one front door with routing and provider abstraction, policy and budgets in one place, shared reliability, caching, tracing and guardrails, and a paved road teams actually want to use. Then it assembles every earlier episode into one architecture.
Takeaway: centralise the things every team gets wrong on their own, but don't build a bottleneck. Adoption is a design requirement.
Who this is for
Backend engineers, architects and tech leads who are being asked to ship LLM features and want to do it without learning every lesson the hard way. You don't need ML experience. If you know what a retry, a queue and an idempotency key are, you have most of the foundation already.
All the chats, orders, traces and prices in the videos are fictional demo data, and charts marked illustrative are worked examples rather than benchmarks.
Watch the full series: https://www.youtube.com/playlist?list=PLOooQo0oCPgU
If you're preparing to move from backend into an AI engineering role, I'm also making a companion series on the interview side of that transition: From Backend to AI Engineer.
I'd genuinely like to hear where you disagree. Which of these trade-offs would you make differently in your system?
