Most RAG tutorials open with "set your OPENAI_API_KEY." This one doesn't need it. In Part 1 I claimed the LLM and embeddings are behind a swappable boundary — "switch providers via config, not code." Part 3 is me cashing that claim: running the entire RAG agent — ingestion, retrieval, the ReAct loop, source citations — on a laptop with zero API keys and no Docker, just Ollama and an embedded Qdrant.
Everything below is real output from an actual run. Including the one thing that broke.
What "offline" actually requires
Three pieces, all local:
- Ollama running two models — one for chat, one for embeddings:
ollama pull qwen3.5:9b # chat / reasoning
ollama pull bge-m3 # embeddings (1024-dim, multilingual)
- Embedded Qdrant — no server, no container. The vector store writes to a local directory.
- A one-line config flip so chat goes to Ollama instead of the gateway:
CHAT_PROVIDER=ollama
That's it. No OPENAI_API_KEY, no docker compose up. The reason this is a flip and not a rewrite is the provider-swap design from Part 1 — let's look at the three factories that make it work.
The embeddings factory — swap by config
# app/llm/embeddings.py
@lru_cache
def get_embeddings() -> Embeddings:
s = get_settings()
provider = s.embedding_provider.lower()
if provider == "ollama":
from langchain_ollama import OllamaEmbeddings
return OllamaEmbeddings(model=s.embedding_model, base_url=s.ollama_url)
if provider == "openai":
from langchain_openai import OpenAIEmbeddings
return OpenAIEmbeddings(base_url=f"{s.litellm_url}/v1",
api_key=s.litellm_key, model=s.embedding_model)
raise ValueError(f"unknown embedding_provider: {s.embedding_provider!r}")
Both branches return the same LangChain Embeddings interface, so the ingestion and retrieval code never knows which one it got. Local dev → Ollama (offline). Production → OpenAI via the gateway. One caveat that matters later: the two providers produce different vector dimensions, so you can't mix vectors ingested with one and queried with the other. More on that in the gotchas.
The vector store — embedded vs. remote, also by config
# app/rag/store.py
@lru_cache
def get_client() -> QdrantClient:
s = get_settings()
if s.qdrant_url:
return QdrantClient(url=s.qdrant_url, api_key=s.qdrant_api_key) # remote (prod)
return QdrantClient(path=s.qdrant_path) # embedded (local)
No QDRANT_URL? You get an embedded client that persists to s.qdrant_path — a plain directory. Set QDRANT_URL in prod and the same code talks to a real Qdrant service. The trade-off of embedded mode: it locks the directory to a single process, which becomes gotcha #2.
Ingestion: docs → chunks → vectors
The ingest script is the whole pipeline in ~30 lines: load files, split them, probe the embedding dimension, create the collection, upsert.
# scripts/ingest.py (trimmed)
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)
chunks = splitter.split_documents(documents)
# probe the embedding dimension so the collection matches the provider
dim = len(get_embeddings().embed_query("probe"))
ensure_collection(dim)
get_vector_store().add_documents(chunks)
The embed_query("probe") trick is worth pausing on: instead of hard-coding 1024 for bge-m3 (or 1536 for OpenAI), it asks the active embedder for one vector and measures it. Swap the provider and the collection is created with the right size automatically.
Running it for real:
$ python scripts/ingest.py --reset
[ingest] source=docs collection=docs embed=ollama:bge-m3
[ingest] 5 documents → 53 chunks
[ingest] embedding dim = 1024
[ingest] done — 53 points in collection
Five markdown files, 53 chunks, 1024-dim vectors from bge-m3, written to the local Qdrant directory. No network calls left the machine.
Running the agent — no server needed
You can hit the FastAPI endpoint, but to see the graph think you can also invoke it directly. Here's a real run, asking about something that lives in the docs:
res = await graph.ainvoke({"messages": [HumanMessage(content=
"How is short-term vs long-term memory implemented in this project?")]})
print([type(m).__name__ for m in res["messages"]])
# ['HumanMessage', 'AIMessage', 'ToolMessage', 'AIMessage']
That message sequence is the ReAct loop, visible in the state:
-
HumanMessage— the question -
AIMessagewithtool_calls=[search_docs(...)]— the model decides to retrieve -
ToolMessage— the retrieved chunks come back -
AIMessage— the final synthesized answer
And the answer itself, generated entirely by a 9B model on the laptop:
Short-term memory: PostgreSQL (PostgresSaver) stores per-thread
conversation state; swappable to Redis (RedisSaver) if needed.
Long-term memory: Zep manages the user's persistent knowledge,
recalled by the app on later turns.
Sources: <doc-a>.md, <doc-b>.md
Grounded in the actual docs, with source attribution, zero API keys. That's the win. Now the part the tutorials skip.
Gotchas (the part that's actually worth reading)
1. The empty synthesis turn — the local model, not the pipeline
On one run, the exact same question produced this:
[1] AIMessage content='' tool_calls=[search_docs(...)] finish_reason='tool_calls'
[2] ToolMessage content='[1] (source: ...) ## memory layers ...' ← retrieval worked
[3] AIMessage content='' tool_calls=[] finish_reason='stop' ← empty answer
Retrieval succeeded. The chunks were right there in step 2. But step 3 — the model's job to read the chunks and answer — came back empty. finish_reason='stop', no tokens, no error. Re-running the same question gave a perfectly good 280-character answer with citations. So it's intermittent: a small local model occasionally produces an empty turn after a tool call.
Two things to take away:
- It's the model, not your graph. The pipeline (routing → retrieval → state) was flawless; the synthesis step just whiffed.
- The
saw_tokenfallback from Part 2 won't save you here — that fallback callsainvokewhen no tokens stream, but hereainvokeis the empty result. The real mitigations are a larger/better tool-tuned local model, or accepting some flakiness as the price of fully offline. Worth knowing before you demo it live.
2. Embedded Qdrant locks the directory
Embedded mode keeps the store in one process. Run the ingest script while the server is up and you'll get a lock error. Order matters: ingest first → let it exit → then start the server. The ingest script even closes the client explicitly to avoid a noisy shutdown traceback.
3. Embedding dimensions must match end to end
bge-m3 is 1024-dim; OpenAI's text-embedding-3-small is 1536. If you ingest with one provider and query with another, the dimensions don't line up and search breaks. Switching embedding_provider means re-ingesting (--reset). The embed_query("probe") dimension check is exactly what keeps the collection honest per provider.
4. The first call is slow
Ollama loads the model into memory on first use. The first request eats that cost; subsequent ones are fast. Don't benchmark the cold start.
Why this matters
You can build, debug, and demo the entire RAG agent — graph, retrieval, citations — on a plane with no wifi. Then, for production, you flip two config values (CHAT_PROVIDER, QDRANT_URL) and the same code talks to a hosted model and a real Qdrant cluster. Part 1 claimed the provider boundary; Part 3 ran on both sides of it.
The flip side is honesty about local models: retrieval is rock-solid, but a 9B model's synthesis step is the weak link, and it'll occasionally hand you an empty answer. Know that going in.
Next: persisting conversation threads with a checkpointer — so the agent remembers across requests — and what that adds to the message log you just saw.
Part 3 of a series on running LangGraph in production. Part 1 · Part 2.
