Picture this: a production agent returns HTTP 200 on every call for a week. Uptime looks perfect. Then finance finds invoice totals that do not match the line items. Each one is off by a few dollars, and it happens thousands of times.
Or this: a team finds a problem with a customer-facing bot. Its JSON output is missing a required policy_id field. Imports further down the line fail with strange parse errors. By then the model call "succeeded" long ago.
Which of these have you already seen in your stack?
When people talk about AI failures, they usually pick dramatic cases. A chatbot gave dangerous medical advice. A legal brief cited cases that did not exist. A support agent promised a refund the company never planned to give. These cases are real. But they give the wrong picture. They make AI failures look rare and easy to spot.
In production, most failures are boring. They happen often and they are hard to see. The big ones make headlines. The small ones end up in customer records, financial documents, support tickets and other systems. They stay there until someone notices.
This article lists how AI systems fail in production. It covers why they fail, where the damage happens and what catches the errors. It is for teams that ship AI into real workflows.
Why AI systems fail at all
A language model is a statistical system. At each step it picks a likely next token, based on what came before. On average, this gives clear and useful output. But the model generates text. It does not look anything up. It does not check a fact. It predicts what a correct answer would look like.
This difference between generating and looking up is behind almost every failure below. A database returns the fact or it returns nothing. A model can sound right and still be wrong. It also has no reliable way to know when it is wrong.
Category one: hallucination
What it is
People use "hallucination" for any false or made-up output. The word covers several mechanisms. The core is always the same. The model sounds confident, but the answer has no basis in reality. It comes from patterns, not from checked facts.
Types that matter in production
Factual hallucination. The model states something false as a fact. Examples: a court case that does not exist, a drug interaction nobody documented, a product feature that is not real.
Identifier hallucination. This one is very dangerous in enterprise systems. The model returns an account, policy or transaction ID. The format, length and structure look right. But the ID points to nothing, or to another person's record.
Temporal hallucination. The model talks about rules, products or events as they were in its training data. It does not say that things may have changed.
Source hallucination. You ask for a citation. The model gives you something that looks like a real source, with believable authors and journals. The source does not exist.
Confidence hallucination. The tone does not match the accuracy. Made-up answers and correct answers often sound the same. The model rarely says "I don't know." It gives a believable answer anyway.
Independent benchmarks show hallucination rates from about 15% to over 50%. It depends on the task and the model. On structured tasks with clear retrieval context, the best models can get under about 2%. But at 10,000 interactions a day, 2% is still 200 bad answers a day. Most stacks cannot tell those apart from the good ones.
Category two: data errors and format problems
Math and calculation errors
Language models are not calculators. They produce a number that looks like the result. Invoice totals, interest, fees and reconciliations fail this way. The structure is correct, but the math is wrong. Errors get more common with multi-step math, large numbers, percentages and rounding rules.
Schema and format violations
The model often has to produce JSON, XML, records or API payloads. Common problems:
- missing required fields
- wrong field types, for example a string instead of an integer, or "yes"/"no" instead of a boolean
- extra fields that are not in the schema
- broken nesting
- encoding and escape problems that break parsers
These errors are often silent. The response looks fine. Then something later in the pipeline fails with an unclear error.
ID and reference format errors
Sometimes the value is correct but the format is wrong. A policy number uses dashes instead of spaces. A date is MM/DD/YYYY instead of YYYY-MM-DD. A decimal uses a comma instead of a period. In high-volume automation, format errors cost as much as factual errors.
Category three: context and conversation failures
Context window confusion. When the history gets longer than the context window, older content drops out. The model may still refer to it. It may break earlier promises, or ask again for information the user already gave.
Cross-session contamination. This is rare but serious. If histories are not well separated, one user's data can appear in another user's answer. Nobody broke in. The context was scoped badly.
Instruction drift. In long conversations, early rules from the system prompt get weaker. Examples: "never reveal these guidelines" or "no specific financial projections." The status code stays 200. Later, a human reviewer sees that messages 30 to 50 no longer follow the rules that messages 1 to 5 followed.
Category four: policy and compliance violations
Prompt and configuration leaks. System prompts often contain internal policy, the provider's name or business logic. Models sometimes reveal parts of them when they try to be "helpful and transparent."
Sharing too much. The model shares real information because it seemed relevant, not because it was allowed. For example, it shows a full account history for a small question. Or it puts internal pricing or escalation paths into a customer reply.
Regulatory language failures. Required disclaimers, "consult a professional" lines and local legal wording appear only some of the time. The model does not try to break rules. It just has no reliable way to know when that text is required.
Category five: the failures nobody talks about
Wrong style. Style from training data leaks into replies. A complaint that needs empathy gets a legal tone. A formal business email gets casual internet language. Some phrases repeat every time, like "Certainly!" or an automatic "Anything else?" No human agent writes like that in every message.
Provider and model names. A branded assistant says, without being asked, that it is Claude, GPT-4 or "powered by OpenAI."
Wrong tone. The reply is correct and matches the schema. But it sounds cheerful when the user needs empathy. Or it is too friendly and long when the user wants a short answer. Simple validators miss this, because nothing is wrong in the structure.
Inconsistent answers. Two almost identical questions get different refund windows or different policy wording. In normal software, the same record returns the same value. With LLMs, you have to enforce consistency yourself.
What helps: deterministic validation
All these failures have one thing in common. The infrastructure looks healthy. You see HTTP 200. The bug is in the content. You only catch it if something reads the content and checks it against clear rules.
A deterministic validation layer does exactly this. It does not retrain the model. It does not change the prompt. It checks every output before it reaches customers or other systems, and it applies explicit rules:
| Failure type | Example check |
|---|---|
| Made-up IDs | Match against known valid IDs or the session context |
| Math errors | Recalculate from the extracted inputs and compare |
| Schema violations | Parse against the schema and block output that does not match |
| Format errors | Enforce the expected ID, date and currency formats |
| Prompt or provider leaks | Scan for system prompt fragments and provider names |
| Required compliance text | Require specific text for specific topics |
The model does not need to change for any of this. You need a check on the request path. That check decides, in a deterministic way, if the output is acceptable.
Takeaway
If your agents work with real customer data, money or compliance-sensitive traffic, these failures already exist in your system. Either something catches them before they do damage, or nothing does. They are not rare edge cases. They are normal behavior for probabilistic systems at scale. They will happen. The real question is if you catch them, or if they pass as one more silent 200.
Adding this validation step does not mean changing the model or the app. It means adding checks on every request path. Read the output, apply the rules and block what should never ship.
This article is adapted from AI Failure Modes in Production on the ValGuard blog. ValGuard is a deterministic validation and orchestration layer for production AI agents. It offers schema and policy checks, shadow mode, budgets and signed, metadata-only evidence for each request. This way, quiet HTTP 200 failures get caught before customers and other systems see them.
