Why RAG gives wrong answers (and how to fix retrieval failures)

In the previous post, we built a RAG system from scratch. Sixty lines of Python. Six onboarding documents chunked, embedded, and searchable. Two questions asked, two correct answers grounded in the actual policy documents.

It worked great. So obviously, let's break it to learn how to fix it.

And then we go deeper than the video had time for: the full toolkit of chunking and retrieval strategies, and when you'd actually reach for each one.

All the code is in my GitHub repo, in the ep06-rag-failures folder. Each failure and its fix are separate files that change exactly one thing, so the difference is the whole story.

Two ways RAG fails

Here's the thing about RAG: when the model gives you a wrong answer, the instinct is to blame the model. But most of the time, the model did exactly what it was told. The problem is what it was given. The retrieval step handed it the wrong evidence. Wrong evidence in, confident wrong answer out.

I asked two new questions against the same system. Both failed, for different reasons at different layers.

Failure 1: Chunks too small

Question: "Can I expense my home office setup during onboarding?"

The system retrieved five chunks. A cross-reference from the expense policy: "See Benefits Guide Section 5.3 for home office stipend details." A section heading with just its intro line: "5.3 Home Office Setup, for hybrid and remote employees." A mention of the ergonomic assessment. The scores were decent, around 0.47. The system found the right documents.

But the actual answer, the $750 stipend on hire, the $250 annual refresh, the $1,200 ergonomic budget, none of it made it into the retrieved chunks. Those numbers sat in a chunk that never cracked the top five.

And the model was honest about it. It said it didn't have specific information about whether you can expense home office setup during onboarding, and that the details sit in Benefits Guide Section 5.3, "which isn't fully shown in the context provided."

This is not hallucination, its incomplete response. Because I (or RAG here) gave it incomplete evidence.

Diagnosis: The paragraph-based split cut the section too short. Right place, incomplete answer.

Failure 2: Ambiguous match

Question: "What's the 90-day rule?"

Five chunks came back from four different documents, all with low, similar scores in the 0.27 to 0.39 range. Nothing stood out, which is exactly the problem. "90 days" shows up in completely different contexts across these policies:

Probation period: a 90-day review with your manager
RRSP matching: starts after 90 days of employment
Expense claims: submit within 90 days
Security access: elevated permissions expire after 90 days

Several different "90-day rules." The embeddings all carry the same phrase. Semantic search can't tell them apart.

The model handled it reasonably. From the chunks it was handed, it surfaced two of them, the security access rule and the probation review, and asked "which context were you asking about?" But think about what happened.

The system searched everything, pulled from four different documents, burned tokens, and the best it could do was ask me to clarify.

With 6 documents, fine. With 6,000, every vague question floods the model with irrelevant context. Costs go up, latency goes up, quality goes down.

Diagnosis: Same words, different meanings. The embeddings can't distinguish them.

The key insight

Neither failure is a model problem. The model did exactly what it was told: answer from the provided context. Both failures are in retrieval. Good news is, retrieval is tunable.

The two failures also live at two different layers.

One is about how you split the documents i.e. chunking. The other is about how you search them i.e. retrieval. Understanding this difference is important, because the fix is different for each.

Fix 1: Heading-based Chunking

The home office problem was a chunking problem. My paragraph split cut sections too short. But these onboarding docs have structure. Headings. Sections.

Heading-based chunking keeps a section together: each heading plus everything under it until the next heading becomes one chunk. "5.3 Home Office Setup" plus the intro line AND the bullet list of dollar amounts, all one piece.

import re

HEADING_RE = re.compile(r"^#{2,6}\s")  # split on ## and deeper, keep the # title attached

def chunk_by_heading(text: str, source: str) -> list[dict]:
    """Each heading + everything beneath it, until the next heading, = one chunk."""
    chunks, current = [], []

    for line in text.split("\n"):
        if HEADING_RE.match(line) and current:
            block = "\n".join(current).strip()
            if block:
                chunks.append({"text": block, "source": source})
            current = [line]
        else:
            current.append(line)

    block = "\n".join(current).strip()
    if block:
        chunks.append({"text": block, "source": source})
    return chunks

Same question again: "Can I expense my home office setup during onboarding?"

This time the retrieved chunk contains the full section. $750 on hire, $250/year refresh, up to $1,200 for the ergonomic assessment, all of it.

The model has complete evidence and gives a complete answer.

The fix wasn't a better model. It was a better split.

The chunking toolkit: pick your split on purpose

Heading-based chunking fixed our first failure. But it isn't a universal answer. If your documents have no headings, it won't help. If a section is enormous, you're back to the "too much context" problem from the context window post.

So let's walk the full menu. Same idea every time: how it splits, and when you'd reach for it.

Fixed-size splitting. Cut every N tokens, done. It's the baseline everyone starts with. It works for uniform text like transcripts or logs. But it splits mid-sentence and mid-paragraph, so answers land right on a boundary and get cut in half. That's the exact failure we just fixed.

Recursive splitting. Try to split on headings first. If a chunk is still too big, split on paragraphs. Still too big? Sentences. It's a fallback chain, not a single rule. Most frameworks use this as their default, and it's a good one when you don't yet know the shape of your data.

Semantic chunking. Instead of splitting on structure, split on meaning. Frameworks like LangChain (SemanticChunker) and LlamaIndex (SemanticSplitterNodeParser) hand you this as a single call. Under the hood it embeds each sentence, compares consecutive sentences, and cuts wherever the similarity drops. The text tells you where the topic changes. It costs more up front because you embed sentence by sentence, but for dense material without clean headings it's often worth it.

Parent-child chunking. Store chunks at two levels. Small chunks make search precise. Large parent chunks give the model complete context. When a small chunk matches, you return its parent to the model. Search small, feed big. You get precise retrieval and full context, at the cost of more plumbing.

Side by side, so you can scan the tradeoffs in one place:

There's no universal right answer. A Chroma study on chunking strategies for retrieval found that how you chunk can matter as much as which embedding model you pick. Two knobs, comparable weight.

A reasonable starting point for most systems: recursive splitting, 256 to 512 token chunks, 10 to 20% overlap. That's a default, not a law. Chunking is a decision you make based on your data shape, not a setting you copy from a tutorial.

And the only way to know if your choice is working is to measure it. Build a set of 20 to 30 test questions with expected answers. Change one thing, rerun, compare. "It feels better" is not a metric.

Fix 2: Metadata filtering for Retrieval

The "90-day rule" problem isn't about chunk size. It's about disambiguation. The embeddings all contain the same phrase, so semantic search alone can't separate them. But we know something useful the search doesn't: each chunk came from a specific document.

The fix is to tag each chunk with its source when you store it, then filter to one source before the similarity search runs. Narrow first, search second. This is called pre-filtering.

# At ingestion, stamp every chunk with a source label
{"text": "...", "source_label": "employee-handbook"}
{"text": "...", "source_label": "expense-policy"}
{"text": "...", "source_label": "benefits-guide"}

# At query time, shrink the candidate set BEFORE scoring
def retrieve(question, chunks, embeddings, top_k=5, source_label=None):
    q_vec = np.array(embed_text(question))

    if source_label:  # the whole fix is this pre-filter
        candidates = [i for i, c in enumerate(chunks)
                      if c.get("source_label") == source_label]
    else:
        candidates = range(len(chunks))

    scored = [
        (i, np.dot(q_vec, embeddings[i]) / (np.linalg.norm(q_vec) * np.linalg.norm(embeddings[i])))
        for i in candidates
    ]
    scored.sort(key=lambda x: x[1], reverse=True)
    return [{**chunks[i], "score": s} for i, s in scored[:top_k]]

Same question, "What's the 90-day rule?", filtered to the employee handbook.

Now it searches 26 of the 185 chunks instead of all of them.

The answer is clean: it's the probationary period. Bi-weekly manager check-ins, no internal transfers, a formal review at the end.

Switch the filter to the expense policy? Same question, different clean answer: submit expense claims within 90 days of the transaction.

Same vague question. A different right answer each time. Because we narrowed the search space before the similarity search even ran.

Who picks the filter?

The obvious catch: I picked the filter by hand. I decided "search the employee handbook." But if a user just asks "What's the 90-day rule?", how does the system know which document to search?

There's a pattern for this called a self-querying retriever. You pass the question to a model first and ask: "Based on this question, which metadata filter should I apply?" The model reads the question, looks at what fields exist, and extracts the filter. "What's the 90-day rule for expenses?" gives back source = expense-policy. If the question is truly ambiguous, it can ask the user to clarify.

This doesn't need a big model. Picking the filter is just a classification task, so the model that picks the filter isn't the model that writes the answer. A small, fast, cheap model reads the question and picks the filter. The big, expensive one only steps in for the final answer. That's the "right model for the right job" point from the model sizing post, applied inside a single pipeline. And it's not just filters. That same small model can handle the other quick decisions too: routing the question, classifying what's being asked, pulling out a name or a date.

Frameworks like LangChain give you this as a building block. Managed services like Amazon Bedrock Knowledge Bases run the filter step as part of the pipeline. They don't swap models behind your back. You still choose which model does what. They handle the wiring: pull the filter out of the question, apply it before the search.

One production note on that source tag, and a distinction worth being precise about. In the demo, the tag lives in a Python dictionary, and we attach it to each chunk while we're chunking. So it's chunk-level metadata. Every piece knows which document it came from because we stamped it during ingestion.

That's not the only place metadata can live. It can also sit on the file itself. If your documents are in Amazon S3, a feature called S3 annotations lets you attach rich, updatable context straight to the object, and it stays queryable.

Here's the difference. Our example adds metadata while chunking, so it's per-chunk. S3 annotations add metadata to the whole file, so every chunk from that file inherits it. File-level context is great for the coarse cut: which document, which source, which category. Chunk-level context is what you need when different chunks from the same file deserve different tags. Most real systems use both. Annotate the file for discovery, tag the chunk for precision.

The retrieval toolkit: search smarter, not harder

Chunking is half the story. How you search those chunks is the other half. Metadata filtering was one tool. Here's the rest of the box, from the ones you'll use daily to the ones you'll reach for when the easy stuff isn't enough.

Start with the basics

Hybrid search. Combine semantic search with keyword search. Semantic finds meaning. Keywords find exact terms. Run both, blend the scores. If I search "90-day probation," the keyword "probation" narrows it even when the embeddings are all similar. This is huge for anything with codes, product names, acronyms, or IDs, where the exact string matters and embeddings blur it. BM25 is the classic algorithm on the keyword side. Most production systems use hybrid as their baseline.

Re-ranking. Retrieve a wide net first, say 20 candidates, with fast search. Then re-score just those 20 with a slower, more precise model called a cross-encoder. Instead of comparing the question and each chunk as separate vectors, the cross-encoder reads them together and judges relevance directly. You keep the speed of vector search for the wide pass, and add accuracy on the short list. Reach for this when precision at the top matters and you can afford one extra step.

Query routing. Instead of searching one big index, route the question to the right sub-index first. "Benefits question? Search benefits docs. Security question? Search security docs." It's metadata filtering where the system picks the filter for you, based on classifying the question. Worth it when you have several distinct corpora that don't overlap much.

Reach for these when the basics aren't enough

HyDE (Hypothetical Document Embeddings). The trick: questions and answers don't look alike. A question is short and interrogative; the passage that answers it is longer and declarative. So HyDE asks a model to write a fake, throwaway answer to the question first, then embeds that and searches with it. The made-up answer looks more like a real document than the question does, so it lands closer to the right chunks. Reach for it when your users ask questions phrased very differently from how your documents are written.

Reciprocal Rank Fusion (RRF). When you run several retrievers (semantic, keyword, maybe HyDE), you get several ranked lists. RRF merges them without needing their scores to be on the same scale. Each list contributes points based on where an item ranks, roughly 1 divided by its position, and you add up the points across lists. Items that rank high in more than one list bubble to the top. It's the simple, robust way to fuse hybrid results.

Query decomposition. Some questions are really several questions in a trenchcoat. "How do parental leave and RRSP matching interact for someone in their first year?" is three lookups. Decomposition breaks the question into sub-questions, retrieves for each, then combines the evidence. Reach for it when questions are multi-part and no single chunk could ever hold the whole answer.

Multi-hop retrieval. Retrieve, read what you found, then retrieve again based on it. "Who approved the remote work exception for the engineering team?" First hop finds the exception and the team. Second hop uses the manager's name you just learned to find the approval. It chains. Reach for it when the answer is a trail of facts, each one pointing to the next.

How to actually choose

Don't reach for the fancy stuff first. Start simple and add only when your test set says you need to:

Get chunking right. Most "bad retrieval" is bad chunking wearing a disguise.
Add metadata filtering or routing to narrow the search space.
Add hybrid search so exact terms stop getting blurred.
Add re-ranking when the right answer is in the results but not at the top.
Only then reach for HyDE, decomposition, or multi-hop, for the specific query shapes that need them.

Every layer adds latency and cost. Add each one because a metric moved, not because a blog post mentioned it.

Scaling beyond a Python list

At scale, you move from a Python list to a vector database. Something like pgvector if you already run Postgres, or Amazon OpenSearch Serverless, or Amazon S3 Vectors if you want vector storage that behaves like S3. Or, if you don't want to own any of it, Amazon Bedrock Knowledge Bases runs the whole pipeline as a managed service: connectors, parsing, chunking, embedding, retrieval. You point it at your data, it does the rest.

One thing to keep in mind: every embedding call costs a fraction of a cent. With six documents, irrelevant. With a million documents, you do the math up front. Same principle we covered in the model sizing post: cost sets the ceiling.

Also worth noting explicitly: Titan Embeddings for search, fast and cheap, that's its job. Claude for generation, smart and conversational, that's its job. Different models for different steps in the same pipeline. You'll see that pattern again when we get to agents.

GraphRAG: when chunks aren't enough

Standard RAG treats every chunk as an independent piece of text. It grabs the closest one and stops. But real documents are connected. A policy names a team. That team has a manager. That manager approved an exception. Each fact points to the next, and those links are information too. Standard RAG just drops them.

GraphRAG keeps the links. It builds a knowledge graph from your documents, mapping how entities connect. When you ask a question, it doesn't just find the closest chunk. It walks the relationships between them.

"Who approved the remote work exception for the engineering team?" Standard RAG struggles because the answer spans three chunks. GraphRAG connects the dots because it stored the relationships explicitly. It's multi-hop retrieval, but the hops are baked into the data structure instead of done at query time.

We're not building this today. But it's worth knowing it exists. The field is moving fast.

Key takeaways

If you're just getting started: When AI gives you a wrong answer from your documents, it's probably not a model problem. It's a search problem. Check what was retrieved. The model answers from whatever evidence it's given. Better evidence, better answers.

If you're more on the builder side: RAG failures decompose into two layers. Chunking is how you split the documents. Retrieval is how you search them. Diagnose which layer broke, fix that layer. Then reach for smarter strategies in order, cheapest first, and let a test set of 20 to 30 queries tell you whether each one actually helped. "It feels better" is not a metric.