Your local RAG isn't slow — it re-reads every document on every question

A user opens a project with nine files in it, types the most obvious question
anyone types at a document app — "what are these documents about?" — and waits.

291 seconds.

Then they ask a second question, about one of those documents, and wait again.
Minutes, not seconds. At that point the app has told them something about
itself, and what it has told them is: this model is slow and probably stupid.

The model was neither. It was reading. It read the entire retrieved corpus,
from token zero, for question one. Then it read it again for question two.

Where the 291 seconds went: prefill, not generation

Setup, for symptom matching: an offline desktop app on llama.cpp, M4 Pro with
24 GB, a 14B at Q5, nine files in the project.

Two throughput numbers, both from my own logs:

generation:  ~8 tokens/second     <- normal for a 14B at Q5 on this box
prefill:   ~100 tokens/second     <- also normal

Neither is a bug. The bug is the ratio, and what got multiplied by it. That one
question assembled a 13,773-token prompt, and the turn as a whole pushed
17,373 prompt tokens through the model once you count the service calls it
makes on the side. At a hundred tokens a second, reading dominated writing by
roughly five to one; the profile came out north of 85% prefill.

Divide those numbers yourself and you'll land a few dozen seconds off. Prefill
throughput sags as the context grows, and one "question" is more than one model
call. The shape is the point, not the arithmetic: the machine spent its evening
reading, and the part the user was waiting for — the answer — was cheap.

I want to be precise about the thing I had gotten wrong for a long time,
because I don't think I'm alone in it. I had indexing. Chunks, embeddings, a
vector store, the whole ritual, run once when files are added. What I assumed
that ritual bought me was that the documents were, in some sense, already
read.

It buys nothing of the kind. Retrieval is a table of contents, not a memory. It
finds the right pieces quickly; it does not make the model read them faster,
and it does not make the model remember having read them. Every question ships
fresh text into the context window and the model chews it from the first token,
at prefill speed, every time. In classical RAG, indexing time and reading time
are separate budgets, and only one of them is ever spent in advance.

Three things came out of the logs before I got to the interesting part.

Finding 1: two retrievers, one prompt, ~9,000 duplicated tokens

I run lexical search by default and vector search behind a flag. On the
machines where both are on, both contribute passages to the prompt. They union
their results, deduplicated by chunk id.

Chunk id. Not content. The two engines index with different chunk boundaries,
so the same paragraph arrives as two different ids with substantially the same
text, and the union happily keeps both.

Roughly 9,000 tokens per question were the same passages, twice. Not once
in a while — on every question, for as long as both engines had been on. Nobody
caught it by reading the prompt, because nobody reads the prompt; it is a wall
of text that scrolls past in a debug log and looks exactly like a wall of text
is supposed to look.

Deduplicating on normalized content instead of id took the question from 291
to 150 seconds. That is the least interesting bug here and it was
worth more seconds than anything I did on purpose that week.

Finding 2: the prefix cache hit 3 tokens out of 17,373

llama.cpp keeps a per-slot prompt cache. Send a prompt that shares a prefix
with the last one that slot saw, and it skips prefill for the shared part. In a
chat app that should be most of the system prompt, most of the time.

Mine, trimmed:

slot update_slots: id  0 | task 412 | n_past = 3, cache_tokens = 3, n_prompt_tokens = 13773

Three tokens. Beginning-of-sequence and a bit of chat template. The cache had
never once helped, on any turn, since the feature existed.

The cause was a helper I'd been pleased with. Before retrieval, a small service
call rewrites the user's question into a better search query — resolving "it"
and "that contract" against the conversation. It's a good feature. It ran with
its own system prompt, and it ran into the same slot as the conversation.

Slot caches key on the longest common prefix. Two different system prompts
diverge at token three. So the service call evicted the conversation's cached
prefix, the conversation's next turn evicted the service call's, and the two
took turns doing this forever. The cache was working exactly as designed. It
was caching a conversation that alternated, every single turn, with a
completely different conversation, in the same chair.

Splitting the slots — a pool for conversations, a pool for service calls, never
shared — moved the hit rate from 0% to 45-48%.

Finding 3: the study queue lived in RAM and died with the process

There was already a background "study" pass in the codebase, meant to
pre-summarize documents after import. It kept its queue in process memory.

Close the app, and every unfinished item is gone. Not retried — gone, with no
record that it had been scheduled. And it only ran while the app was open and
otherwise idle, which on a desktop app is a narrow and unreliable window.

Net effect: summaries were essentially never present when a question arrived,
so every question fell back to reading document bodies. A background job that
doesn't survive a restart is a background job that never finishes, because the
user closes the window constantly and does not consider this an unusual thing
to do.

Study at indexing: pay the minutes once per file, not once per question

Here is the reframe the whole thing turned on. Prefill is not a cost you can
optimize away — a 14B reading N tokens has to read N tokens. It is a cost you
can move. The question is whether the model reads a document at question
time, while a human watches a spinner, or at indexing time, when nobody is
waiting.

Three pieces, in order of how cheap they are.

The document passport, ~200 tokens, no model involved

For each file, a small structured record: document type, parties, dates,
amounts, page count, and a table of contents extracted from heading patterns
with a regular expression. Not with the model — with a regex, at import, in
milliseconds.

That's about 200 tokens per document. For a nine-file project, "what are these
documents about?" now has a 1,800-token answer surface where it used to
have 13,773.

The honest limit: heading extraction works on documents that have formatting
and fails flat on an unstructured wall of text. Those fall through to the pass
below.

Map-reduce summaries, in the background

Per file: chunk it, summarize each chunk, summarize the summaries. This costs
exactly the minutes you were paying before — the model still reads the whole
document — except it costs them once, on import, and never again.

The queue rules matter more than the summarization prompt:

• it yields to live questions, always; a user typing outranks it instantly
• it goes silent on battery
• it checkpoints per chunk and resumes after a restart, because see Finding 3
• it stores state in the database, not in a variable

The user does pay for this. They pay in fan noise, once per file, at a moment
when they are not staring at a progress bar.

A cascading router that picks the cheapest lane that can answer

• Broad question ("what are these documents about", "summarize the project") — answer from passports plus summaries. 1-2k tokens.
• Specific question ("what's the termination clause in the lease") — vector retrieval, narrow set of chunks, as before but smaller.
• Explicit deep read — full bodies, no shortcuts, and the user knows they asked for the slow path because they had to say so.

Routing is the risky part and I'd rather name the risk than sell around it.
Misroute a specific question into the digest lane and you answer from a summary
that dropped the exact number the user wanted, confidently and wrongly. Two
mitigations, both boring: the router is biased to escalate — if a question
mentions a term that appears in a passport's table of contents but not in the
summary, it goes to chunks — and the app shows which lane answered, so a thin
answer has an obvious "go read it properly" next to it.

What the numbers look like now

102 seconds is not a good number. It is a much better number, and it is an
honest one: a 14B on a laptop reading five thousand tokens has to read five
thousand tokens, and no amount of architecture argues with that.

The exact-match cache is the cheapest line in the table and it exists because
of a behavior I did not predict. Users re-ask the identical question. They
close the app, come back, and type the same words to see whether it's still
right. Hashing the normalized question plus the resolved context set and
keeping the answer turns that into a second.

Why 100% prefix-cache hits are not on the table

The reusable part of a prompt is the part that doesn't change. System prompt
and passports are stable, so they cache. Retrieved excerpts change with the
question — by construction, since changing them is the entire job of retrieval.

You could force them to cache. Retrieve once per conversation, freeze the
context, and every subsequent turn shares a long identical prefix. I sat with
that for a while and turned it down. The second question in a conversation is
usually about something the first question didn't retrieve; freezing the
context buys cache hits and pays for them in wrong answers. 45-48% is what the
stable prefix is genuinely worth in this layout, and I'd rather report that
number than a better one I bought with accuracy.

One thing that is free: order the prompt by volatility. Stable first
(system prompt, passports), volatile last (excerpts, then the question). Get
that backwards and your hit rate is zero no matter how much of the prompt is
technically stable.

I went and checked what everyone else does

Before claiming any of this was novel I looked at what the neighbors ship: LM
Studio, AnythingLLM, Jan, GPT4All, Open WebUI with Ollama.

All of them chunk and embed at index time. Not one of them precomputes
per-document digests or summaries. The best of them keep a prompt cache, which
helps with the system prompt and does nothing for the retrieved half.

Which means that on "what are these documents about?" — the single most common
opening question a human asks a document app, the one they type before they
type anything else — every one of these re-reads the corpus at full prefill
cost, every time.

I'm supposed to call that a gap in the market. It's really a gap in the default
architecture: the reference RAG design does retrieval at question time and
nothing at index time except embeddings, everybody copied it faithfully, and
the copy is correct. It's just that "correct" and "the user waited five
minutes" are compatible states.

If you're building a local RAG

Measure prefill against generation before you optimize anything. If 85% of
your wall clock is reading, a faster sampler and a smaller quant are noise. The
engine prints both numbers; find the line.

Deduplicate context sources by content, not by id. Two retrievers with
different chunk boundaries will hand you the same paragraph twice and neither
will look wrong in isolation. This was 9,000 tokens a question in my app.

Give service LLM calls their own KV slot. Query rewriting, classification,
title generation — anything with its own system prompt sharing a slot with the
conversation will zero your prefix cache and the logs will still say the cache
is enabled.

Precompute passports and summaries at index time. A ~200-token structured
passport per document, with headings pulled by regex rather than by the model,
answers most broad questions on its own and costs no GPU at all.

Put the study queue in the database. In-memory queues on a desktop app do
not survive contact with users, who close windows. Checkpoint per chunk, resume
on launch, yield to live traffic, stop on battery.

Cache the literal repeats. People re-ask the same question verbatim more
than you'd think. Hash the normalized question plus the resolved context set.

If you want to know whether any of this is worth your afternoon, the arithmetic
is short enough to run before you commit to it: file size, prefill speed,
number of questions, and it tells you how many seconds of pure reading you have
already signed up for. That's how_long_will_my_rag_wait.py — https://github.com/JackYU96/rag-rereads-every-question — in the repo next
to this post — no dependencies, one file, bring your own numbers.

The last time I chased a number like this, the KV cache itself turned out to be
eating it: "V cache quantization requires flash_attn" — the llama.cpp error
that quietly halves your context
window.
Different layer, same lesson. The engine will tell you where your seconds went.
It tells you in a log line that scrolls past at startup, in a field you have
never once looked at.