Part 4 of 8 — RAG Article Series
Previous: How RAG Works: The Complete Pipeline (Part 3)
Chunking Is a Design Decision
Part 3 showed that ingestion splits documents into chunks before embedding them. Most tutorials pick a chunk size — 512 tokens is popular — and move on. That works when every document looks the same. TechNova's documents do not look the same — and that difference is where chunking decisions start to matter.
The firmware changelog is a flat list of version entries. The troubleshooting guide has numbered procedures under section headers. The product specs page has a comparison table. Each document has a different internal structure, and each will break differently under the same chunking strategy. Chunking is not a setting you toggle. It depends on what your documents actually look like. You can inspect these files in the companion repository — Part 5 walks through each one in detail.
Fixed-Size, Recursive, and Semantic Chunking
Fixed-size chunking splits every N tokens regardless of content. It is fast, predictable, and easy to debug. It is also blind to structure. A 512-token window will cut TechNova's Bluetooth pairing procedure between step 3 and step 4 if that is where the token count falls. The chunk boundary does not know it is splitting a procedure.
Here is that procedure from TechNova's troubleshooting guide (the full file is in the companion repository at data/troubleshooting-guide.md):
- Open Settings → Bluetooth on your device.
- Forget "WH-1000" from saved devices.
- On the WH-1000, hold the power button for 7 seconds until the LED flashes blue.
- Select "WH-1000" when it appears in your device's Bluetooth list.
- Wait for "Connected" confirmation before playing audio.
A 512-token chunker does not know these five steps belong together. It sees a stream of tokens and splits by size. If the size boundary falls after step 3, one chunk gets steps 1–3 (open settings, forget the device, enter pairing mode) and the other gets steps 4–5 (select the device, confirm the connection). Steps 1–3 disconnect your headphones. Steps 4–5 reconnect them. A user who asks "How do I fix Bluetooth disconnection?" may get only the first chunk — an answer that tells them how to tear down their Bluetooth connection but never tells them how to restore it.
Fixed-size chunking works best for documents with consistent, uniform structure — the firmware changelog, where every entry is a self-contained version note.
Recursive chunking splits by document structure: first by section, then by paragraph, then by sentence if the section is still too long. It respects the boundaries your documents already have. TechNova's troubleshooting guide, with its H2 headers and numbered steps, splits cleanly along section lines. Each chunk is a complete procedure or topic. This is the practical default for most teams because most documents have some structural markers — headers, paragraphs, list boundaries — and recursive splitting uses them.
Semantic chunking uses embeddings to detect where the topic shifts. Instead of relying on structural markers, it measures the similarity between consecutive sentences and cuts where the meaning changes. This can help with documents that genuinely lack structural markers — long unstructured transcripts where topics shift mid-paragraph with no headers or section breaks. But it is not the first tool to reach for when documents have mixed formats. TechNova's product specs (see data/product-specs.html in the companion repository) have tables and prose — that is a parsing problem, not a chunking problem. If you feed raw HTML into a text splitter, table rows get separated from their column headers, and a chunk might contain "8 hours" with no indication of which product or spec that refers to. A structure-aware parser followed by recursive chunking usually handles it. Semantic chunking is more expensive, harder to debug, and can produce inconsistent results. Treat it as an escalation when recursive chunking is not enough, not as the default for anything that looks complex.
Start simple. Parse the document well first — handle tables, headers, and lists before you think about chunking strategy. Then use recursive chunking as your default. If chunk boundaries are splitting procedures or separating facts from their context, add overlap. Only consider semantic chunking when the document genuinely lacks structural markers and evaluation shows recursive splitting is not working well enough.
There are additional chunking patterns — hierarchical (parent/child) chunking, contextual chunking, and others — that become relevant once your baseline pipeline is running. We cover these in Part 8.
Late Chunking: A Different Order
There is a newer approach worth knowing about. Instead of chunking first and embedding each chunk on its own, late chunking flips the order: embed the full document first, so every token carries context from its surroundings, then split. Each chunk remembers pronouns, headers, and references that pointed elsewhere in the document.
A 2025 study found trade-offs: contextual retrieval keeps more semantic coherence but costs more compute, while late chunking is cheaper but can lose some relevance. We cover standard chunking first because it is the baseline you need to understand before optimizing. Late chunking is something you evaluate once that baseline is working — not where you start.
The Overlap Question
Chunks without overlap lose information at boundaries. The Bluetooth procedure above shows the cost: steps 1–3 in one chunk, steps 4–5 in the next. Neither chunk contains the full procedure. The retriever returns one of them, and the model generates an incomplete answer.
Overlap means repeating the last two to three sentences of each chunk at the start of the next. Both chunks now contain step 3, so whichever the retriever returns has enough context to connect to the rest of the procedure. The trade-off is real but manageable: more storage, and the possibility that both overlapping chunks are retrieved, producing near-duplicate context. In practice, a two-sentence overlap is a reasonable default that most teams start with and rarely need to change.
This connects to a pattern you will see throughout this series. When a RAG system produces vague or hedging answers — "The return policy may vary depending on the product" instead of a specific number — that is usually a chunking problem. The chunks were too broad, too generic, or split in a way that diluted the specific fact the user needed. You see the symptom in the output, but the fix is upstream in the ingestion pipeline. In Part 7, we will build a complete diagnostic framework around symptoms like this one.
Retrieval — Keyword, Semantic, or Hybrid
Chunking determines what the retriever can find. The retrieval approach determines how it searches. There are three options, and they have different strengths.
Term-Based Retrieval (BM25)
BM25 matches on exact terms. When a user asks "WH-1000 return policy," BM25 finds every chunk that contains those words and scores them by how distinctive those terms are within the corpus. It is fast, requires no embedding model, and excels at precise, specific queries where the user knows the right vocabulary.
It fails when the user does not use the same words the documents use. "Can I send back my headphones?" contains neither "return" nor "policy." BM25 returns nothing useful. The information exists in the index. The query just does not match the terms.
Embedding-Based Retrieval
Embedding-based retrieval matches on meaning, not terms. "Can I send back my headphones?" and "Return policy: 15 days from date of delivery" share no significant words, but they mean similar things. The embedding model sees that similarity, and the retriever finds the right chunk.
The weakness is on the other side. "WH-1000 battery life" and "WH-500 battery life" may embed to nearly identical vectors because the embedding model treats both as "battery life for a headphone product." If the model does not understand that WH-1000 and WH-500 are distinct products with different specs, it may return the wrong product's chunk. Semantic retrieval is flexible but loses precision on exact distinctions.
Hybrid Search and Reciprocal Rank Fusion
Run both. BM25 and vector search execute in parallel on the same query, each producing a ranked list. Reciprocal Rank Fusion merges the two lists by rank position — not raw score — so both approaches contribute equally.
The result: "WH-1000 return policy" retrieves well because BM25 catches the exact terms. "Can I send back my headphones?" retrieves well because vector search catches the meaning. Neither approach alone handles both queries. Together, they cover each other's gaps.
Hybrid search is the practical default for production RAG systems. It adds implementation complexity — two retrieval passes instead of one — but it eliminates the most common retrieval failures. Most teams that start with vector-only search migrate to hybrid once they see the edge cases that exact-term matching would have caught.
One Question, Three Configurations
To see why these decisions matter, consider a single question against TechNova's troubleshooting guide: "My WH-1000 keeps disconnecting from Bluetooth. What should I do?"
Configuration A: Fixed-size chunking (512 tokens), vector-only retrieval. The troubleshooting guide's Bluetooth section has five numbered steps. The 512-token boundary falls between step 3 and step 4. The retriever returns the chunk containing steps 1–3. The model generates an answer that starts the procedure but stops mid-way: "First, go to Settings and forget the device. Then re-enable Bluetooth and…" The answer trails off or the model fills in a plausible but wrong next step. The reader gets a partial procedure that looks complete.
Configuration B: Recursive chunking with overlap, vector-only retrieval. The recursive chunker keeps all five steps in one chunk. The model generates the full answer. But the query says "keeps disconnecting" instead of "Bluetooth troubleshooting," and the vector-only retriever sometimes returns a firmware changelog entry about a Bluetooth stability fix instead — the embeddings are close enough to confuse it.
Configuration C: Recursive chunking with overlap, hybrid retrieval (BM25 + vector + RRF). The chunks are the same as Configuration B. But now BM25 also runs and catches "WH-1000" and "Bluetooth" as exact terms, anchoring the retrieval to the right product's troubleshooting section. The firmware changelog entry drops in rank because it talks about a fix, not a troubleshooting procedure. The model receives the correct, complete procedure and generates the full answer.
Same question. Three configurations. Three different answers. The model was the same every time. What changed was the chunking and retrieval decisions made before the model ever saw the query.
Reranking — The Second Pass That Matters
The first retrieval pass — whether BM25, vector search, or hybrid — is optimized for speed. It returns the top candidates quickly, but "most similar" is not always "most relevant." A chunk about the WH-1000's Bluetooth specifications might rank highly for a question about Bluetooth pairing issues, because the terms and concepts overlap. But the user needs the troubleshooting procedure, not the spec sheet.
A reranker is a cross-encoder model that reads each candidate chunk alongside the original query and scores how well the chunk actually answers the question. It is slower and more expensive than the first pass — which is why it only runs on the top 10–20 candidates, not the entire index. The first pass gets candidates fast. The second pass sorts them by actual relevance. Together, they produce better results than either alone.
When to add reranking: when your retrieval results are in the right neighborhood but not in the right order. The right chunk is often in the top 10 results but rarely in position 1. A reranker pushes the best answers to the top. It is one of the highest-value, lowest-effort improvements teams make after the initial build.
Evaluate Before You Optimize
A team swaps their embedding model from a general-purpose model to a domain-specific one, expecting retrieval to improve. They redeploy. Customer satisfaction drops. It takes two weeks to trace the problem: the new model embeds TechNova's product codes differently, and queries about the WH-1000 now occasionally retrieve WH-500 content. The model change made retrieval worse, and nobody measured before or after.
If you cannot measure retrieval quality, you cannot improve it. Every decision in this article — chunking strategy, retrieval approach, reranking — is an experiment. Without measurement, you are guessing.
Two metrics matter most at this stage. Context precision: of the chunks you retrieved, how many were actually relevant to the question? If 3 of 5 returned chunks are useful, precision is 60%. Context recall: of all the relevant chunks in your knowledge base, how many did you retrieve? If the answer requires 2 chunks and you found both, recall is 100%. Precision tells you how much noise is in your retrieval. Recall tells you how much signal you are missing.
Start small: 20–50 queries with known-good answers and the chunks that should be retrieved. Run retrieval, measure precision and recall, compare before and after every change. Part 7 builds a full diagnostic framework on top of this foundation.
One more lever worth knowing about: tagging chunks with metadata like product ID, document type, or version number lets you filter before retrieval, so the retriever only searches the relevant slice of your index. We will revisit this in Part 8 when we cover production concerns.
Three Takeaways
Chunking is a design decision shaped by your documents, not a fixed default. Different documents create different failure modes. Start with recursive chunking and escalate only when evaluation shows you need to.
Hybrid retrieval (keyword + semantic) is the practical default for production systems. BM25 catches exact terms. Embeddings catch meaning. Together, they cover each other's gaps.
If you cannot measure retrieval quality, you cannot improve it. Evaluate first. Measure before and after every change. Part 7 shows you how.
The engineering decisions are clear. Now it is time to build. You have the pipeline model from Part 3 and the decision framework from this article. Part 5 puts them together: a working RAG system, built from scratch, using TechNova's documents.
Next: Build a RAG System In Practice
More in the next part — I'd love to hear your thoughts on this one.
