Your RAG pipeline is bad at docs because your chunker splits code fences

Here is the chunking code from more or less every RAG tutorial published in the last two years:

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
)
chunks = splitter.split_documents(docs)

It works fine on prose. Run it over a documentation site and it quietly does four destructive
things, none of which throw an error, and all of which show up later as "the assistant gives
vague answers about our API."

I spent a while building a docs-specific ingestion pipeline and most of the work turned out to be
in these four places. None of it is glamorous. All of it moved retrieval quality more than
swapping embedding models did.

1. It cuts code fences in half

A 1,000-character window lands mid-code-block constantly. You get one chunk ending in:

```python
def authenticate(client_id, client_secret):
    resp = requests.post(TOKEN_URL, data={

and the next chunk starting with the rest of the dict and a closing fence that now has no opener.

Two things go wrong. The obvious one is that neither chunk contains a runnable example, so the
model retrieves half a function and confabulates the other half. The subtler one is that a
half-open fence poisons everything downstream — every markdown renderer, and most models,
treat the remainder of that chunk as code. Your carefully written prose about token expiry is now
inside a Python block as far as the model is concerned.

The fix is that code blocks and tables can only split at line and row boundaries, and if a
split does happen the fence has to be reopened with its language tag and the table header has to
be repeated. And overlap must never bisect a fence — the overlap window is where this bug hides,
because the chunk itself looks fine and only the neighbour is broken.

2. It throws away where the chunk came from

This is the one that costs the most retrieval quality for the least effort to fix.

A sliding window gives you a chunk that reads:

You must include the state parameter and verify it on return. Tokens expire after 3600
seconds.

Embed that and ask "how long do OAuth tokens last?" and it may or may not come back, because
nothing in the text says OAuth, or authentication, or which product this is. The words that would
have matched are in an <h2> four hundred characters up the page.

Documentation is a tree and chunks should inherit their path. Every chunk in a docs pipeline
should open with its heading path:

Authentication > Authorization flows > OAuth 2.0

### OAuth 2.0

You must include the `state` parameter and verify it on return...

Now the chunk is independently answerable. It carries its own context, it embeds near the
question people actually ask, and when you show sources in the UI you have a breadcrumb to
display instead of a bare URL.

Keep the un-prefixed version in a separate field too (rawText), because when you feed retrieved
chunks to the model for generation you often want the clean body without the breadcrumb noise
repeated ten times in the prompt.

3. chunk_size=1000 isn't 1000 of anything you care about

RecursiveCharacterTextSplitter counts characters. Your embedding model has a token limit.
The ratio between them varies enormously with content:

So a 1,000-character chunk is ~250 tokens of prose but can be ~500 tokens of code. If you sized
your window against a model limit using a character count, your code chunks are silently getting
truncated at the embedding step, and truncation at the embedding step is invisible — no error, no
warning, just a vector for the first 60% of the chunk.

Count real BPE tokens with the actual tokenizer (tiktoken / gpt-tokenizer, cl100k_base or
o200k_base depending on your model). It is one dependency and it removes a whole class of
"why is retrieval worse on the API reference pages" investigations.

4. It has no idea your docs site has four versions of every page

Documentation sites are duplicate factories. Versioned docs (/v2/, /v3/, /latest/),
locale variants, print views, and framework-generated index pages mean a naive crawl of a
500-page docs site can yield 1,800 chunks where 600 would do.

Exact-hash dedupe catches byte-identical pages and gets you maybe half of it. The rest needs
near-duplicate detection — SimHash with banded lookup is cheap and works well here.

But there's a trap. Naive near-duplicate detection will happily collapse:

Install on Linux — Run ./configure && make && make install.

Install on Windows — Run ./configure && make && make install.

Identical bodies, completely different answers to "how do I install this on Windows?" So
similarity matching has to be scoped to the heading: two chunks are only candidates for
deduplication if their heading path matches too.

Doing all four without writing a crawler

I ended up packaging this as an Apify actor —
Docs-to-RAG Pipeline Builder — because I
was rebuilding the same thing for every project and the crawl half is more annoying than it looks.

Minimal run:

{
  "startUrls": [{ "url": "https://docs.example.com" }],
  "maxCrawlPages": 500
}

That returns chunks that already respect all four rules above. Each one looks like:

{
  "id": "9f2c1a77b0e34d15",
  "url": "https://docs.example.com/api/authentication",
  "anchorUrl": "https://docs.example.com/api/authentication#oauth-2-0",
  "pageTitle": "Authentication",
  "breadcrumb": ["Authentication", "Authorization flows"],
  "heading": "OAuth 2.0",
  "headingLevel": 3,
  "chunkIndex": 2,
  "chunkCount": 5,
  "text": "Authentication > Authorization flows > OAuth 2.0\n\n### OAuth 2.0\n\n...",
  "rawText": "### OAuth 2.0\n\n...",
  "tokenCount": 612,
  "contentHash": "1b0f...",
  "generator": "mkdocs-material",
  "extractor": "profile:mkdocs-material",
  "crawledAt": "2026-08-24T10:14:22.104Z"
}

Note anchorUrl. Deep-linking a citation to the exact heading rather than the page top is a
two-line change that makes a support bot feel dramatically more trustworthy, and almost nobody
does it.

Two design decisions worth stealing even if you build your own

Crawl in two passes, not one. Rendering every page in Chromium is slow and expensive;
rendering none of them breaks on every client-side-rendered docs site. So: fetch over plain HTTP
first, score the extracted content, and escalate only the pages that fail the gate — JS shells,
suspiciously thin pages — to a browser. renderingMode: "auto" with escalateBelowWords: 60
does this. On a typical docs site the large majority of pages never touch a browser.

Let extraction strategies compete instead of picking one. Docs sites are built by generators,
and generators have known DOM shapes. So run several extractors — an explicit
mainContentSelector if the user gave one, a profile matched to the detected generator (MkDocs,
Docusaurus, Sphinx, Starlight, VitePress and friends), a general readability-style extractor, and
a link-density heuristic falling back to <body> — score all of their outputs, and take the
winner. A single hardcoded selector is a silent failure waiting for the day the docs site
upgrades its theme; it doesn't error, it just starts returning navigation sidebars as content.

The run report tells you which extractor won on which pages, which is how you find out your
selector went stale.

Embeddings without an API bill

embeddingProvider has three settings, and two of them cost nothing:

{
  "startUrls": [{ "url": "https://docs.example.com" }],
  "embeddingProvider": "cloudflare-worker",
  "workerUrl": "https://docs-to-rag-worker.you.workers.dev",
  "workerApiKey": "<API_TOKEN>"
}

You deploy a small Worker to your own Cloudflare account and it runs @cf/baai/bge-m3 (1024
dims) on Workers AI. The free tier covers roughly 9 million tokens a day — a 500-page site
chunked at 800 tokens uses about 0.5% of one day's allowance. Set embeddingProvider: "local"
instead and it runs Xenova/all-MiniLM-L6-v2 (384 dims) on-device with no network calls after
the model downloads.

Both are meaningfully worse than a frontier embedding model on hard retrieval. Both are entirely
adequate for "answer questions about our docs", and being free changes how willing you are to
re-embed after changing your chunking — which you will, several times.

The artifacts nobody asks for and everybody ends up using

Alongside the dataset it writes to the key-value store:

• chunks.jsonl — the embedding-ready records
• corpus.md — the whole site as one readable markdown file
• llms.txt and llms-full.txt — the llms.txt site index format
• run-report.json — extractor wins, escalations, failures

corpus.md is the one I use most and expected least. When retrieval gives a wrong answer, being
able to grep the entire corpus in one file tells you in five seconds whether the content was
missing from the crawl or present-but-not-retrieved. Those two failures look identical from the
chat UI and have completely different fixes.

Measure it on your own corpus, because I can't measure it on yours

Retrieval quality claims that aren't measured on your data aren't worth much, and mine aren't
either. The good news is that this particular A/B is cheap to run, and free if you use local
embeddings.

Build the same corpus twice:

1. Baseline — RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200) over raw HTML converted to text
2. Structure-aware — heading-recursive chunking, breadcrumb prefixes on, dedupe on

Then write 20–30 questions you already know the answer to. This is the part people skip and
it's the only part that matters. Take them from your actual support inbox or your team's Slack —
questions real people asked, not questions you invented while looking at the docs, which are
biased toward the phrasing the docs already use.

Three numbers to compare:

• Chunk count. The dedupe delta is usually the first thing that jumps out on a versioned docs site. Fewer chunks at equal coverage is a straight win on embedding cost and retrieval noise.
• Hit rate @5 — for each question, is the chunk containing the answer in the top five? This is the number that predicts whether your assistant feels good to use.
• Whole-example rate. Of the retrieved chunks containing code, how many contain a complete, runnable block rather than a fragment? This is the one the sliding window loses badly, and it's worth counting separately because hit-rate alone hides it.

If structure-aware chunking doesn't beat the baseline on your corpus, that is a real and useful
finding — it probably means your docs are flatter than you thought, and your effort belongs in
the retriever or in reranking instead. Measure before you commit to any of this.

Where I'd push back on myself

Structure-aware chunking is not free. It's a lot more code than eight lines of LangChain, and on
a corpus that isn't structured — support-ticket exports, transcripts, PDFs of scanned contracts —
none of it helps, because there are no headings to be aware of. The sliding window is genuinely
the right default there.

The claim is narrower than "sliding windows are bad": documentation is one of the most
structured corpora you will ever index, and throwing that structure away at the chunking step is
the single most common own-goal in RAG pipelines over docs. If you're indexing something else,
ignore most of this and go tune your retriever instead.

The actor is Docs-to-RAG Pipeline Builder
(MIT-licensed, pay-per-event, runs with a single startUrls entry). If you'd rather build your
own, the four rules at the top of this post are the ones I'd implement first — in that order.