RAG Chunking Explained: How to Choose the Right Chunk Size and Strategy

Imagine I give you a whole pizza and say:

“Eat it.”

You look at it and think, Sure, I can eat it.

But now imagine I give you the same pizza without cutting it.

Can you eat it comfortably?

Not really.

You would probably cut it into smaller pieces first.

And that is exactly what we do with large documents before giving them to an AI system.

A 200-page PDF, a 50,000-line documentation file, or a huge company knowledge base is useful to us as a complete document.

But for a Retrieval-Augmented Generation (RAG) system, giving the entire document to the retrieval system is usually not the best idea.

So we break it down.

Document
   ↓
Chunks
   ↓
Embeddings
   ↓
Vector Database
   ↓
Retrieval
   ↓
LLM
   ↓
Answer

That process of breaking a large document into smaller, meaningful pieces is called Chunking.

And here is the interesting part:

Chunking is not simply “splitting text into 500 characters.”

It is a retrieval design decision.

The wrong chunk can make the right information difficult to retrieve.

Index

1. Why Does Chunking Matter?
2. How Chunking Fits Into RAG
3. The Chunk Size Trap
4. Chunking Strategies
5. Chunk Size and Chunk Overlap
6. Implementing Chunking with LangChain
7. How to Choose the Right Chunking Strategy
8. How to Evaluate Chunking
9. A Practical Chunking Evaluation Experiment
10. Common Chunking Mistakes
11. My Practical Chunking Workflow
12. Final Takeaway

1. Why Does Chunking Matter?

Let's say we have this document:

Company Employee Handbook

Chapter 1: Leave Policy
Employees receive 20 paid leaves every year...

Chapter 2: Work From Home
Employees can work remotely two days per week...

Chapter 3: Insurance
Employees are eligible for health insurance...

Now a user asks:

“How many days can I work from home?”

We don't need the entire employee handbook.

We need the small section containing the Work From Home Policy.

So ideally:

                 DOCUMENT
                     │
        ┌────────────┼────────────┐
        ↓            ↓            ↓
     Chunk 1      Chunk 2      Chunk 3
        │            │            │
   Leave Policy   WFH Policy   Insurance
                     │
                     ↓
               Relevant Chunk

The goal is simple:

Retrieve the smallest useful piece of information without destroying its meaning.

The Two Ways Chunking Goes Wrong

There are two opposite mistakes.

Chunks are too large

Imagine:

┌──────────────────────────────┐
│ Leave Policy                 │
│ Work From Home               │
│ Insurance                    │
│ Salary                       │
│ Performance Review           │
│ Travel Policy                │
│ ...                          │
└──────────────────────────────┘

You ask:

“How many leaves do I get?”

The answer exists somewhere inside the chunk.

But so does a lot of unrelated information.

The retriever has brought the answer plus noise.

Chunks are too small

Now imagine:

Chunk 1:
Employees are eligible for

Chunk 2:
20 days of paid leave

Chunk 3:
every financial year.

The original document contained a complete thought.

Our chunking destroyed it.

The information exists.

But the meaning is fragmented.

This gives us the first important principle:

Chunking is a balance between context and precision.

Too large:

More context
    ↓
More noise

Too small:

More precision
    ↓
Less context

The goal is somewhere in between.

2. How Chunking Fits Into RAG

A typical RAG pipeline looks like this:

                    ORIGINAL DOCUMENT
                           │
                           ▼
                    ┌─────────────┐
                    │   Chunking  │
                    └──────┬──────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
           Chunk 1      Chunk 2      Chunk 3
              │            │            │
              ▼            ▼            ▼
          Embedding     Embedding    Embedding
              │            │            │
              └────────────┼────────────┘
                           ▼
                    Vector Database
                           │
                           │ User Query
                           ▼
                     Query Embedding
                           │
                           ▼
                       Retrieval
                           │
                           ▼
                    Relevant Chunks
                           │
                           ▼
                          LLM
                           │
                           ▼
                        Answer

There is one important thing to notice:

Chunking decides what the embedding model actually gets to understand.

If you change the chunks, you change the embeddings.

If you change the embeddings, you can change retrieval.

And if retrieval changes, the context given to the LLM changes.

So chunking is not just preprocessing.

It is part of your retrieval architecture.

3. The Chunk Size Trap

This is probably one of the most misunderstood parts of chunking.

Someone might ask:

“My LLM supports 128K tokens. Should I create 8K-token chunks?”

Not necessarily.

Your LLM's context window is only one part of the equation.

Your chunk size depends on several things:

                     CHUNK SIZE
                         │
        ┌────────────────┼────────────────┐
        ↓                ↓                ↓
 Embedding Model     Retrieval          LLM
  Input Limit       Requirements    Context Budget
        │                │                │
        └────────────────┼────────────────┘
                         │
                         ▼
                  Practical Chunk Size

And there are two more factors that matter:

• Nature of your documents
• Nature of your questions

So the decision becomes:

Document Structure
       +
Embedding Model Limit
       +
Retrieval Requirements
       +
LLM Context Budget
       +
Question Type
       ↓
Candidate Chunk Sizes
       ↓
Evaluation
       ↓
Final Chunk Size

This is why there is no universal:

chunk_size = 500

number.

Don't start with “What chunk size does everyone use?” Start with “What information am I trying to retrieve?”

4. Chunking Strategies

There is no universally best chunking strategy.

A legal contract, a Markdown documentation page, and a Python codebase don't contain information in the same structure.

So why would we split them in exactly the same way?

LangChain provides different text splitters, and its documentation recommends RecursiveCharacterTextSplitter as a strong starting point for generic text.

Let's group the important approaches.

4.1 Fixed-Size Chunking

The simplest approach:

Every 1000 characters → new chunk

Example:

from langchain_text_splitters import CharacterTextSplitter

splitter = CharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=100
)

chunks = splitter.split_text(text)

The advantage?

Simple and predictable.

The problem?

It doesn't necessarily care about meaning.

It can split:

at the end of a paragraph

or:

in the middle of a sentence.

So fixed-size splitting is useful for simple, predictable data, but it is rarely the only strategy worth considering.

4.2 Recursive Character Chunking

This is usually the best starting point for general text.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)

chunks = splitter.split_text(text)

Conceptually, the splitter tries to preserve larger natural boundaries before breaking the text further:

Paragraph
    ↓
Newline
    ↓
Space
    ↓
Character

So instead of immediately cutting through a sentence, it tries to preserve meaningful blocks first.

Document
   │
   ├── Paragraph
   │      │
   │      └── fits? → keep together
   │
   ├── Paragraph
   │      │
   │      └── too large?
   │              ↓
   │          split further
   │
   └── ...

That's why it is such a useful generic starting point.

When you don't know where to start, start simple. Then measure.

4.3 Token-Based Chunking

Characters and tokens are not the same thing.

For example:

1000 characters ≠ 1000 tokens

The exact relationship depends on the tokenizer, language, and text.

If your downstream system has strict token constraints, token-aware splitting becomes useful.

from langchain_text_splitters import TokenTextSplitter

splitter = TokenTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)

chunks = splitter.split_text(text)

Token-based splitting becomes particularly useful when you need precise control over how much tokenized content enters the embedding or downstream pipeline.

4.4 Structure-Aware Chunking

Sometimes the document already tells us how it should be chunked.

Consider Markdown:

# Authentication

## Login

Information about login...

## Password Reset

Information about resetting passwords...

# Payments

## Refunds

Information about refunds...

Why destroy that structure?

Instead, preserve it.

The same idea applies to:

• Markdown
• HTML
• JSON
• XML
• source code
• legal documents
• technical documentation

For example, a chunk can retain:

Document: API Documentation
Section: Authentication
Subsection: Password Reset

Content:
To reset your password...

Now the chunk contains both:

information + context

That's much more useful for retrieval.

4.5 Code Chunking

Code is not prose.

Consider:

class PaymentService:

    def process_payment(self):
        ...

Randomly splitting every 500 characters can easily separate:

class
   ↓
method
   ↓
implementation

which destroys useful relationships.

For code, you want to preserve programming structure where possible:

Repository
    ↓
Class
    ↓
Method
    ↓
Logical Block

LangChain provides language-aware splitting for multiple programming languages.

So for code:

Respect the syntax before respecting the character count.

4.6 Semantic Chunking

Now we move from:

“Where should I split the characters?”

to:

“Where does the meaning change?”

Consider:

The company was founded in 1998.
It started with five employees.

The company launched its first product in 2001.
Revenue crossed $10M in 2005.

There is a natural semantic transition between these ideas.

Semantic chunking attempts to identify those changes rather than blindly following fixed character boundaries.

This can be useful for highly topic-driven documents.

But there is a trade-off.

It can introduce:

• additional computation
• additional complexity
• additional tuning

So don't automatically assume:

Semantic = Better

Sometimes:

Simple + predictable + evaluated > sophisticated + unevaluated.

5. Chunk Size and Chunk Overlap

Now we reach the famous:

chunk_size=1000
chunk_overlap=200

But what do these numbers actually mean?

Chunk Size

Chunk size defines how much content goes into one chunk.

For example:

chunk_size = 1000

means the splitter attempts to create chunks around that size according to the unit it uses.

That unit could be:

• characters
• tokens
• another model-specific measure

For example, RecursiveCharacterTextSplitter measures characters by default.

Chunk Overlap

Suppose the document says:

Machine learning models require training data.

The quality of this data directly affects model performance.

Good data leads to better generalization.

If we split aggressively:

Chunk 1:
Machine learning models require training data.

Chunk 2:
The quality of this data directly affects model performance.

The relationship between the two chunks can become weaker.

Overlap creates a bridge.

Chunk 1
┌───────────────────────────────┐
│ Machine learning models...    │
│ Training data affects...      │
└───────────────┬───────────────┘
                │
             overlap
                │
                ▼
        ┌───────────────────────────────┐
Chunk 2 │ Training data affects...      │
        │ Good data leads to...         │
        └───────────────────────────────┘

So:

Overlap protects context at the boundary.

But more overlap is not automatically better.

Too much overlap means:

More overlap
     ↓
More chunks
     ↓
More embeddings
     ↓
More storage
     ↓
More duplicate retrieval
     ↓
More context repetition
     ↓
Higher cost

Use overlap when it helps preserve meaning across boundaries.

6. Implementing Chunking with LangChain

Let's build a simple example.

Install the splitter package:

pip install -U langchain-text-splitters

Then:

from langchain_text_splitters import RecursiveCharacterTextSplitter

text = """
Chunking is important for Retrieval Augmented Generation.

Large documents are difficult to retrieve efficiently.

By breaking documents into smaller meaningful pieces,
we can retrieve only the information required to answer
a user's question.
"""

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)

chunks = splitter.split_text(text)

for i, chunk in enumerate(chunks):
    print(f"Chunk {i}")
    print(chunk)
    print("-" * 50)

The flow is:

Original Text
      │
      ▼
RecursiveCharacterTextSplitter
      │
 ┌────┼────┐
 ↓    ↓    ↓
C1   C2   C3
 │    │    │
 └────┼────┘
      ↓
 Embeddings

For token-controlled pipelines:

from langchain_text_splitters import TokenTextSplitter

splitter = TokenTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)

chunks = splitter.split_text(text)

And for structured documents, choose the splitter that understands that structure rather than flattening everything into plain text.

7. How to Choose the Right Chunking Strategy

Here's the question that actually matters:

Which chunking method should I use?

Don't choose based on popularity.

Choose based on your data.

Think about it like this:

                  What is my data?
                         │
        ┌────────────────┼────────────────┐
        ↓                ↓                ↓
     Plain Text       Structured         Code
        │                │                │
        ↓                ↓                ↓
    Recursive        Structure-        Language-
                     aware              aware
        │                │                │
        └────────────────┼────────────────┘
                         ↓
                Check Embedding Model
                         ↓
                  Choose Candidates
                         ↓
                     Evaluate

And that last step is important.

Choosing a chunking strategy is a hypothesis.

Evaluation tells you whether the hypothesis was correct.

8. How to Evaluate Chunking

This is where many chunking tutorials stop.

They shouldn't.

Because the real question isn't:

“Did my document split successfully?”

It is:

“Did splitting improve retrieval?”

A chunk can look perfectly reasonable to a human and still perform badly in a retrieval system.

So we need an evaluation dataset.

Step 1: Create Real Questions

Suppose we're building an employee-policy chatbot.

Create questions such as:

Q1: How many annual leaves does an employee get?

Q2: How many days can employees work remotely?

Q3: What is the maternity leave duration?

Q4: What happens to unused leaves?

Q5: What is the resignation notice period?

For each question, know where the answer actually exists.

For example:

Question:
How many annual leaves does an employee get?

Expected Source:
Employee Handbook
→ Leave Policy
→ Section 2.1

Now we have something measurable.

9. Retrieval Metrics

Suppose the correct chunk is:

Chunk 17

Our retriever returns:

Top 5:

Chunk 91
Chunk 43
Chunk 17   ← Correct
Chunk 52
Chunk 8

The correct chunk appeared in the top five.

That's useful.

Now we can measure different aspects of retrieval quality.

9.1 Recall@K

Recall@K asks:

Did the relevant chunk appear anywhere in the top K results?

For example:

Top 5:

Chunk 91
Chunk 43
Chunk 17 ← Relevant
Chunk 52
Chunk 8

The relevant chunk is present.

Therefore:

Recall@5 = 1

or 100% for this query.

Across many queries:

Recall@5 =

Queries where relevant information
appeared in top 5
───────────────────────────────────
Total queries

So if:

90 / 100

queries retrieved the correct information within the top 5:

Recall@5 = 90%

In simple terms:

Recall asks: “Did I find it?”

9.2 Precision@K

Precision@K asks a different question:

“How many of the retrieved results were actually relevant?”

Suppose:

Top 5:

Chunk 17 → Relevant
Chunk 21 → Relevant
Chunk 42 → Irrelevant
Chunk 63 → Irrelevant
Chunk 91 → Irrelevant

Then:

Precision@5 = 2 / 5
            = 40%

So:

Recall → Did I find the answer?

Precision → How much irrelevant information did I retrieve?

This distinction becomes especially important when large chunks contain multiple unrelated topics.

9.3 MRR — Mean Reciprocal Rank

Now imagine two systems.

System A

Top 5:

1. Correct ✓
2. Wrong
3. Wrong
4. Wrong
5. Wrong

System B

Top 5:

1. Wrong
2. Wrong
3. Correct ✓
4. Wrong
5. Wrong

Both systems found the correct answer within the top 5.

So their Recall@5 is the same.

But are they equally good?

Not really.

System A put the correct result first.

That's where MRR — Mean Reciprocal Rank becomes useful.

For one query:

MRR = 1 / rank of first relevant result

So:

Correct at rank 1
→ 1 / 1
→ 1.0

Correct at rank 2
→ 1 / 2
→ 0.5

Correct at rank 3
→ 1 / 3
→ 0.33

Correct at rank 5
→ 1 / 5
→ 0.20

For multiple queries, we take the average of these reciprocal ranks.

For example:

Query 1 → Correct at #1 → 1.00
Query 2 → Correct at #2 → 0.50
Query 3 → Correct at #3 → 0.33

MRR = (1.00 + 0.50 + 0.33) / 3
    ≈ 0.61

So:

Recall tells you whether you found the answer. MRR tells you how high you ranked the first correct answer.

This is particularly useful when the order of retrieved results matters.

9.4 Don't Evaluate Only Retrieval

A RAG system has multiple layers:

             User Question
                    │
                    ▼
                 Retrieval
                    │
                    ▼
              Retrieved Chunks
                    │
                    ▼
                   LLM
                    │
                    ▼
                  Answer

Therefore, evaluate both retrieval and final answers.

Retrieval Quality

Did we retrieve the right information?

Useful metrics:

• Recall@K
• Precision@K
• MRR

Answer Quality

Did the model actually use that information correctly?

Useful dimensions:

Answer Correctness

Did the model produce the correct answer?

Answer Relevance

Did the answer actually address the user's question?

Faithfulness / Groundedness

Is the answer supported by the retrieved context?

This distinction matters.

Good retrieval does not automatically mean a good answer.

And:

A good-looking answer does not automatically mean good retrieval.

10. A Practical Chunking Evaluation Experiment

Let's say we have:

1,000 documents
200 evaluation questions

We test three configurations.

Experiment 1

Recursive
chunk_size = 256
overlap = 32

Experiment 2

Recursive
chunk_size = 512
overlap = 64

Experiment 3

Recursive
chunk_size = 1024
overlap = 128

Now measure:

Which one wins?

Probably:

512 / 64

Why?

The 1024 configuration has slightly higher Recall@5.

But it retrieves significantly more irrelevant information and ranks the relevant information less effectively.

So the final answer quality also drops.

This gives us another important principle:

The largest chunk is not necessarily the best chunk.

And:

The goal isn't to maximize one metric. It's to find the best trade-off for your application.

11. Metadata: The Missing Piece

Let's say our chunk is:

Employees can work remotely two days per week.

That's useful.

But this is better:

Document: Employee Handbook
Section: Work From Home
Department: HR
Year: 2026

Content:
Employees can work remotely two days per week.

Why?

Because the content tells us what the information says.

Metadata tells us where it came from.

Metadata can also enable filtering:

department = HR
document = Employee Handbook
year = 2026

Then semantic retrieval can happen over a much more relevant subset.

So:

Good chunking tells you what the text says. Good metadata tells you where it belongs.

12. Common Chunking Mistakes

❌ 1. Using an arbitrary chunk size

Tutorial says 500
        ↓
I use 500

That's not a strategy.

Test multiple values against your actual data.

❌ 2. Ignoring the embedding model

Your LLM may support 128K tokens.

Your embedding model has its own input constraints.

Always check:

Embedding model
      ↓
Input/token limit
      ↓
Tokenizer
      ↓
Actual chunk size

❌ 3. Using the same strategy for every document

A legal contract and a Python codebase are not the same thing.

Use the structure already present in the data.

❌ 4. Using too much overlap

More overlap doesn't automatically mean more context.

It can mean:

More duplication
      ↓
More embeddings
      ↓
More storage
      ↓
More retrieval noise
      ↓
Higher cost

❌ 5. Destroying tables and structure

Imagine:

| Product | Price | Discount |
|---------|-------|----------|
| A       | 100   | 10%      |
| B       | 200   | 15%      |

If you blindly split this structure, you can lose relationships between columns and values.

For structured data:

Preserve structure before splitting aggressively.

13. My Practical Chunking Workflow

If I were starting a new RAG system tomorrow, I wouldn't immediately build a complicated semantic chunking pipeline.

I'd start simple.

Step 1 — Understand the Data

PDF?
Markdown?
HTML?
Code?
Legal documents?
Tables?

Step 2 — Preserve Natural Structure

Document
   ↓
Headers
   ↓
Sections
   ↓
Paragraphs
   ↓
Sentences

Don't destroy structure that already exists.

Step 3 — Check the Embedding Model

Ask:

What is the model's input limit?

What tokenizer does it use?

How does it behave with longer inputs?

Remember:

The embedding model is part of the chunk-size decision.

Step 4 — Choose an Initial Strategy

For example:

Plain text
    ↓
Recursive

Markdown
    ↓
Header-aware

Code
    ↓
Language-aware

Highly topic-driven
    ↓
Semantic

Step 5 — Test Multiple Chunk Sizes

For example:

256
512
768
1024

with sensible overlap values.

Don't treat these as universal recommendations.

They are simply candidate configurations to test.

Step 6 — Build a Real Evaluation Dataset

Create questions based on what your users actually ask.

Question
   ↓
Expected Source
   ↓
Expected Information

Step 7 — Measure Retrieval

Track:

Recall@K
Precision@K
MRR

Step 8 — Measure Final Answers

Track:

Answer Correctness
Answer Relevance
Faithfulness / Groundedness

And where useful, also track:

Latency
Token Usage
Embedding Cost
Storage Cost

Step 9 — Pick the Simplest Winner

Suppose:

Recursive Chunking

performs almost as well as:

Semantic Chunking

but is cheaper, faster, and easier to maintain.

Then:

Take the simpler solution.

The best chunking strategy isn't the most complicated one. It's the one that performs well on your data.

14. The Complete Chunking Decision Framework

Put everything together:

                         START
                           │
                           ▼
                    Understand Data
                           │
                           ▼
                 Preserve Data Structure
                           │
                           ▼
                 Check Embedding Model
                           │
                           ▼
                Understand User Questions
                           │
                           ▼
                  Choose Initial Strategy
                           │
                           ▼
                  Choose Chunk Size Range
                           │
                           ▼
                     Choose Overlap
                           │
                           ▼
                 Build Evaluation Dataset
                           │
                           ▼
                  Run Retrieval Tests
                           │
             ┌─────────────┼─────────────┐
             ↓             ↓             ↓
         Recall@K      Precision@K      MRR
             │             │             │
             └─────────────┼─────────────┘
                           ↓
                   Evaluate RAG Answers
                           │
                           ▼
             Correctness / Relevance /
                    Groundedness
                           │
                           ▼
                    Compare Results
                           │
                           ▼
                  Tune and Repeat
                           │
                           ▼
                       Production

The important thing is that chunking doesn't end when the document is split.

It ends when you know those chunks are helping your retrieval system.

15. Final Takeaway

Let's go back to our pizza.

You don't eat a whole pizza in one bite.

You cut it.

But you also don't cut it into 1000 tiny pieces.

Because then you have created another problem.

Chunking works the same way.

Too Large
    ↓
Too Much Noise

Too Small
    ↓
Lost Context

Just Right
    ↓
Better Retrieval
    ↓
Better Context
    ↓
Better Answers

And the "just right" size is not a universal number.

It depends on:

your document structure + your questions + your embedding model + your retrieval strategy + your LLM context budget.

So don't ask:

“What is the best chunk size?”

Ask:

“What is the best chunk size for my data, my embedding model, and my retrieval problem?”

Because ultimately:

Chunking isn't about making documents smaller.
It's about making knowledge retrievable.

And that's the real job of chunking.

Cut the document enough to retrieve what matters — but not so much that you lose why it matters.