I Tested 4 RAG Chunking Strategies Everyone Recommends. 2 Were Quietly Broken.

If you've researched how to build a RAG system, you've seen the advice: use recursive splitting, or better yet semantic chunking, or better yet hierarchical chunking for the tricky cases. Every article explains why each one should work. Almost none of them show what happens when you actually point these techniques at real documents and check the output.

I did that. Took 140 real pages from Microsoft's Azure Service Bus documentation, ran them through four chunking strategies — recursive splitting, semantic chunking, hierarchical (parent-child) chunking, and Contextual Retrieval — and looked, chunk by chunk, at what came out the other side.

Setup: fully local stack — LM Studio for inference, nomic-embed-text for embeddings, LangChain + ChromaDB for orchestration.

Two of the four had genuine bugs. One had a subtle flaw that persisted even after I tried to prompt my way out of it. Here's what that taught me, with the numbers, the fix, and the code.

All four strategies follow the same basic pattern — load the docs, hand them to a splitter, get chunks back:

from langchain_text_splitters import RecursiveCharacterTextSplitter
# swap this import for SemanticChunker, ParentDocumentRetriever's splitters,
# or the Contextual Retrieval prompt step below — everything else stays the same

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=450,
    chunk_overlap=0,
)
chunks = splitter.split_documents(documents)

What differs between the four is what came out the other side. That's what this post is actually about.

The chunk that wasn't a chunk

The numbers: 1,330 chunks from 140 documents, average 266.2 tokens, 69% landed in the healthy 200-399 token range, max never exceeded the 450-token budget. Solid on paper.

Then I pulled a sample chunk to eyeball the output, expecting a paragraph of Azure documentation. Instead I got this:

title: Advanced Features in Azure Service Bus Messaging
description: This article provides a high-level overview...
ms.topic: concept-article

Metadata. Every single document in the corpus starts with a YAML header like this, and the very first "chunk" the system produced was just that header — no actual content, nothing a user's question could ever meaningfully match against.

No chunking algorithm on earth fixes this. It's not a splitting problem, it's a data problem — and it's exactly the kind of thing that never shows up in a benchmark paper, because benchmark papers use pre-cleaned datasets. Real corpora aren't pre-cleaned.

FRONTMATTER_PATTERN = re.compile(r"^---\n.*?\n---\n", re.DOTALL)
doc.page_content = FRONTMATTER_PATTERN.sub("", doc.page_content, count=1)

Chunk count before the fix: 1,366. After: 1,330. This is the first thing I'd check on any real client corpus — production RAG failures trace back to ungoverned source data more often than to chunking algorithm choice.

The 5,000-token "chunk"

The numbers, same corpus, semantic chunking instead of recursive:

Semantic chunking has a good pitch: instead of cutting text at arbitrary character counts, cut it wherever the meaning shifts. Sounds strictly better than mechanical splitting. Those min/max numbers say otherwise.

One document — a technical reference page listing various messaging protocol operations — came out as a single chunk. Not a paragraph. Not a page. The entire document, nearly 5,000 tokens, because the algorithm never found a strong enough topic shift to justify a cut. Reference documentation reads as "the same topic" to a similarity algorithm even when a human would clearly want it broken into digestible pieces.

Alongside it: 4 chunks containing literally nothing — empty strings, silently sitting in what would have become the search index. And the healthy-range rate tells the rest of the story: only 21.2% of semantic chunks landed in the 200-399 token range, against 69.0% for plain recursive splitting on the same corpus.

Neither failure is visible if you only check "did chunking run successfully" and "how many chunks did I get." Both are only visible if you check the extremes, not just the middle of the distribution.

chunks = [c for c in chunks if c.page_content.strip()]

for chunk in chunks:
    if len(encoding.encode(chunk.page_content)) > MAX_CHUNK_SIZE:
        final_chunks.extend(fallback_splitter.split_documents([chunk]))
    else:
        final_chunks.append(chunk)

After the fix: max dropped from 4,972 to 999 tokens, and the healthy-range rate more than doubled to 46.2%. The underlying lesson holds regardless: semantic chunking can fail badly on structurally repetitive content, and reference-style documentation is full of it.

The right answer, ranked dead last

The numbers: query "what are message sessions used for?" against a hierarchical (parent-child) index returned 4 results, each 2,300–4,400 characters — complete, well-formed sections, nothing came back as an orphaned fragment. Full-context return worked exactly as designed.

Here's what actually came back, in order:

1. Message sequencing and timestamps — a related but different feature. Similar-sounding, wrong concept.
2. An SDK/API reference table — mentions session-related method names, but it's a code reference, not an explanation.
3. Request-queue routing with header parameters — not about sessions at all.
4. "Enable FIFO with Sessions" — opens with "Azure Service Bus sessions enable joint and ordered processing of unbounded sequences of related messages. Use sessions in first in, first out (FIFO) and request-response patterns." That's a direct, textbook answer to the query. It came back last.

The best-matching document in the entire corpus for this question wasn't just outranked — it was buried behind three progressively less relevant results. Dense embedding similarity latched onto surface-level word overlap ("sessions," "sequencing," "FIFO") instead of which document actually answered the question.

This one stung a little, because hierarchical chunking is supposed to solve exactly this class of problem: search on small, precise pieces of text, but hand back the full surrounding section so nothing gets returned context-free. It did that — every result was complete, nothing was an orphaned fragment. It just didn't fix ranking.

Worth noting: the frontmatter bug from earlier shows up here too — the raw preview of ranks 1 and 4 both start with the YAML title/description block, the exact same data-hygiene issue that broke the very first recursive chunk.

retriever = ParentDocumentRetriever(
    vectorstore=vector_store,
    docstore=docstore,
    parent_splitter=parent_splitter,   # 1000 tokens — what the LLM sees
    child_splitter=child_splitter,     # 200 tokens — what gets searched
)
retriever.add_documents(documents)
results = retriever.invoke("what are message sessions used for?")

The lesson isn't "hierarchical chunking doesn't work." It's that completeness and correctness of ranking are two different problems, solved by two different techniques, and fixing one tells you nothing about whether you've fixed the other. Completeness needed parent-child chunking. Ranking needs hybrid search (BM25 + dense), reranking, or metadata-aware filtering — a different fix entirely. A lot of RAG advice conflates the two.

The context that leaked

The numbers: across 3 test documents, the LLM generated 13 topic statements describing chunk 1 of each. Checked against the actual source text: 12 correct, 1 leaked — a point about avoiding stored credentials via OAuth 2.0 that actually belonged to the next chunk over, not the one it was describing.

The technique — Contextual Retrieval — asks an LLM to write a short blurb situating each chunk within its source document, so an isolated piece of text doesn't lose its meaning once separated from its neighbors:

prompt = CONTEXT_PROMPT.invoke({"document": full_document, "chunk": chunk.page_content})
generated_context = chat_model.invoke(prompt).content
chunk.page_content = generated_context + "\n\n" + chunk.page_content

I tightened the prompt, told the model explicitly not to describe content from elsewhere in the document. The leak got smaller — 1 out of 13, not 1 out of 5 — but it didn't disappear.

That's not a bug I could code my way out of — it's structural. The technique works by showing the model the whole document for context. That same visibility is what let it blur a boundary. You can't fully remove the risk without removing the thing that makes the technique useful in the first place. Knowing that trade-off exists — and being honest about it instead of claiming a technique is flawless — is worth more than pretending it isn't there.

Why this is the actual point

None of these are exotic failures. They're the kind of thing that ships quietly in a lot of RAG systems, because "it ran without an error" and "it looks fine in a quick demo" get mistaken for "it works." A system that returns a metadata header instead of an answer, or ranks the wrong document first, doesn't crash — it just quietly gives worse answers, and nobody notices until a user complains.

The fix isn't a smarter algorithm. It's a habit: don't trust a technique's reputation, verify it against your own data, and specifically go looking at the edge cases — the shortest result, the longest result, the ones that don't fit the average. That's what caught every failure in this post. Not one of them showed up in the headline numbers.

If there's a single thing worth taking from this: the strategy that wins on a benchmark isn't the strategy that's right for your documents. The only way to know which one is right for yours is to actually test it — and actually look.

What's the weirdest thing you've found when you actually inspected your own RAG chunks? Drop it below.