Last time, we saw how a model's working memory, its context window, can get overloaded, causing long conversations to drift. This time we tackle the most infamous side effect of that same predictive engine: the model confidently inventing facts.
When a language model tells you that the Eiffel Tower is in Rome, it isn't lying. It's doing exactly what it was built to do: predict the next plausible word. Hallucination is not a separate failure mode. It's the same mechanism that makes the model fluent, creative, and useful, just operating where plausibility and truth have drifted apart.
Even a perfectly trained, well-calibrated model must hallucinate on rare facts. That's not an engineering oversight. It's a statistical inevitability baked into the training objective. Once you see why, the whole phenomenon stops feeling mysterious and starts looking like a predictable property of the system.
Why does a model that's so good at predicting words sometimes get facts completely wrong?
The model is not a knowledge base. It's a next-token prediction machine. When you ask it a question, it doesn't look up an answer. It generates a continuation that would be probable in the text it was trained on. If the most probable continuation is factually wrong, the model will confidently produce it.
Think of a chef who has only ever read cookbooks. She can write a flawless recipe for a dish she's never tasted. If the cookbooks contain a typo that calls for salt instead of sugar, her recipe will be confidently wrong. She isn't lying. She's reproducing the pattern she learned.
This is exactly how a language model works. During training, it sees trillions of words from the internet, books, and articles. It learns to minimize the difference between its predictions and the actual next tokens in that data. The objective is purely statistical: maximize the probability of the observed text. There is no separate signal for truth. The model never learns that "vaccines cause autism" is false; it only learns that this phrase appears in certain contexts alongside other phrases. When a prompt echoes those contexts, the model's distribution can assign high probability to the false statement source.
So hallucination happens whenever the model's learned distribution over continuations diverges from the distribution of factually correct answers. The model is optimizing for "what sounds right," not "what is right." That's the core of it.
What does the training data have to do with hallucinations?
The training data is the only window the model has onto the world. If a fact appears rarely, or appears in conflicting versions, the model cannot reliably learn it. This is not a flaw in the data cleaning pipeline. It's a fundamental limit of statistical learning.
Imagine a chatbot asked for a recipe for "Lemon Velvet Cake." Suppose the training data contains exactly one blog post with that recipe, and the post accidentally swaps baking powder for baking soda. The model has no other examples to learn from. To it, that single instance is the entire distribution. If the model is calibrated, meaning its probabilities match real-world frequencies, then it must assign some non-zero chance to the wrong ingredient. On rare facts, that chance translates directly into hallucination.
Theoretical work confirms this. A 2023 paper proved that any calibrated language model must hallucinate on "arbitrary" facts that appear only once in training, at a rate tied to the fraction of such singleton facts source. The model cannot distinguish the true fact from a plausible alternative because it has only seen one example. To avoid hallucination, it would have to say "I don't know." But standard training penalizes non-answers just as harshly as wrong answers. So the model learns to guess.
{
"type": "bar",
"title": "Hallucination rate vs. training frequency",
"caption": "Illustrative: Facts appearing only once in training data are almost certain to be hallucinated. Even at higher frequencies, a small error rate remains.",
"data": [
{
"label": "1 occurrence",
"value": 95
},
{
"label": "10 occurrences",
"value": 60
},
{
"label": "100 occurrences",
"value": 20
},
{
"label": "1000 occurrences",
"value": 5
}
]
}
This also explains why models struggle with recent events, niche topics, or personalized questions like "What did I eat for lunch yesterday?" The training data simply doesn't contain enough signal. The model falls back on patterns from similar contexts, and those patterns may not match reality.
How does the way models are trained make them more likely to confabulate?
During training, the model always sees the correct previous words when predicting the next one. This is called teacher forcing. At inference time, the model must use its own predictions as context. That mismatch is called exposure bias, and it's a major source of hallucination cascades.
Picture a student driver who has only ever ridden with a perfect instructor. The instructor always corrects the wheel before the car drifts. The student never experiences a small mistake and never learns to recover from it. On the real road, a tiny wobble can spiral into a full lane departure because the student has no recovery skill.
For a language model, the first token it generates might be slightly off. Maybe it picks a less common word that still fits. Now that word becomes part of the context for the next prediction. The model has never seen a context that includes its own slightly-off choice during training. So it drifts further. A few tokens later, the output is fluent but completely unmoored from the original intent source.
graph TD
A["Teacher forcing (training)"] --> B["Sees correct previous tokens"]
B --> C["Predicts next token well"]
D["Autoregressive (inference)"] --> E["Sees own generated tokens"]
E --> F["Small initial error"]
F --> G["Errors compound over time"]This is why a chatbot asked to draft a professional email might start with a slightly too casual greeting, then slide into an entirely wrong tone. The first token choice triggers a cascade. The model isn't "deciding" to be unprofessional. It's following its own prior output into a region of text space where the training signal was weak.
Longer generations and higher sampling temperatures both amplify exposure bias. More randomness means more early deviations, which compound. That's one reason why creative writing tasks produce more hallucinations than short factual queries.
Why does the model sound so confident even when it's wrong?
The model's internal state often signals uncertainty. But nothing in its training rewards expressing that uncertainty. Standard benchmarks score answers as right or wrong. A hedged "I think maybe it's X" is treated the same as a wrong answer if X is false. So the model learns to hide its doubt behind a confident surface.
You can see this when you ask a chatbot to summarize a long article that exceeds its context window. The model can't see the middle sections. Instead of saying "I don't have that part," it will often invent plausible-sounding details that fit the surrounding context. It's not trying to deceive you. It's producing the most probable continuation given the incomplete information, and the most probable continuation is a complete, fluent summary.
Researchers have developed ways to measure the uncertainty the model is hiding. One method, semantic entropy, samples multiple answers to the same question and clusters them by meaning. If the model gives semantically diverse answers, its uncertainty is high. High semantic entropy strongly correlates with hallucinated content source. The model "knows" it's uncertain, but it doesn't tell you.
Some newer techniques, like DoLa, exploit the fact that factual knowledge tends to concentrate in specific transformer layers. By contrasting the outputs of deeper and shallower layers during decoding, they can nudge the model toward more truthful tokens without retraining source. These methods work because they tap into the model's own internal signals of factuality, signals that the standard generation process ignores.
Is hallucination a bug or a feature?
It's neither. It's an emergent property of the same mechanism that makes models work. The model's one job is to produce text that is a plausible continuation of the prompt. When plausibility aligns with truth, you get a correct answer. When it doesn't, you get a hallucination. The model isn't switching modes. It's the same prediction engine operating in different regions of the data distribution.
Think back to Part 1: the model predicts the next word. That's it. Everything else, reasoning, creativity, factual recall, is a byproduct of that single trick performed at scale. Hallucination is just the byproduct showing its seams. You can't have the fluent, helpful assistant without also having the occasional confident confabulation, because both come from the same underlying process.
This perspective shifts the engineering challenge. You don't "fix" hallucination like a bug. You design systems around it: retrieval-augmented generation to ground answers in external documents, uncertainty estimators to flag risky outputs, and training incentives that reward appropriate abstention. The model will always be a prediction engine. The goal is to align the contexts where it predicts with the contexts where prediction equals truth.
| Mechanism | What it does | Why it causes hallucinations |
|---|---|---|
| Maximum likelihood training | Optimizes for next-token accuracy on training text | No truth signal; plausible falsehoods get high probability |
| Rare facts (singletons) | Facts seen only once in training | Model cannot statistically distinguish truth from alternatives |
| Teacher forcing | Trains on perfect prefixes only | Model never learns to recover from its own errors; error cascades |
| Unidirectional attention | Processes left-to-right | Model commits early; cannot revise based on later context |
| Benchmark incentives | Reward any answer over "I don't know" | Model learns to guess confidently even when uncertain |
Frequently Asked Questions
Q: Can prompt engineering eliminate hallucinations?
Good prompts can reduce prompt-induced hallucinations by making the task clearer and constraining the output format. But they cannot fix intrinsic hallucinations caused by missing training data or exposure bias. The model's underlying distribution remains unchanged. Prompt engineering is a mitigation, not a cure.
Q: Why do models hallucinate more on long outputs?
Longer outputs give exposure bias more room to compound. Each token is conditioned on the model's own previous tokens, which may already contain small errors. Over many steps, the context drifts further from the training distribution, increasing the chance of fluent but incorrect text.
Q: Does retrieval-augmented generation (RAG) stop hallucinations?
RAG grounds the model in external documents, which greatly reduces hallucinations caused by outdated or missing training data. But the model can still ignore the retrieved context or confabulate details not present in the documents. RAG reduces the problem; it doesn't eliminate it.
Q: Can a model ever "know" it's hallucinating?
Internally, the model often carries signals of uncertainty, as shown by semantic entropy probes. But it has no built-in mechanism to act on those signals unless explicitly trained to do so. Without special training or decoding, it will produce the most probable token even when internally uncertain.
Q: Will future models stop hallucinating?
As training data and architectures improve, hallucination rates on common facts will drop. But the statistical limits around rare facts and the inherent tension between fluency and truth mean some level of hallucination is likely permanent. The goal is to make it manageable, not to eliminate it entirely.
Test yourself
You're building a customer support chatbot that answers questions about your company's internal policies. The model occasionally invents plausible but incorrect policy details when the question is slightly ambiguous. You've already improved the prompts. What's one architectural change you could make that directly addresses the root cause of these hallucinations, and why would it help?
Answer: Add a retrieval step that fetches the exact policy document paragraphs relevant to the question and prepends them to the prompt (RAG). This works because the hallucinations are likely caused by the model falling back on generic patterns from its training data when the prompt is ambiguous. By injecting the ground-truth policy text into the context window, you shift the model's distribution: the most probable continuation given that specific context is now a faithful paraphrase or quote, not a confabulation. This doesn't fix the model's internal tendency to guess, but it changes the input so that guessing isn't necessary. The model can simply attend to the provided text, much like it does when summarizing a visible article.
If this kind of breakdown, how real systems actually work under the hood, is what you're after, subscribe to Internals Decoded at internalsdecoded.com. Next time, we'll look at how models can be steered to be more helpful and honest without breaking their core prediction engine.
Sources
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Calibrated Language Models Must Hallucinate
- Exposure Bias versus Self-Recovery in Autoregressive Text Generation
- Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation
- DoLa: Decoding by Contrasting Layers Improves Factuality
Originally published at Internals Decoded. AI internals, explained conversationally.
