Why the pipeline, not the model, is usually to blame
If your LLM feature starts hallucinating or your inference bill doubles overnight, the model is rarely the root cause. In production RAG systems the usual culprits are missing operational practices: weak retrieval, no reranker, no prompt/version control, poor eval coverage, and blunt cost controls. Treating RAG as “embed → nearest neighbors → prompt” is how teams discover outages, bad answers, and runaway spend.
This article gives a compact, practical LLMOps for RAG systems checklist I use with engineering teams to ship safer, cheaper features in 2–4 sprints.
The 5-step checklist (what to implement and why)
1) Hybrid retrieval (dense + sparse)
Why: Dense embeddings find semantic matches; sparse BM25 preserves exact anchors (IDs, dates, SKUs). Together they cover complementary failure modes and raise recall@k by ~15–30% in benchmarks.
How: Run both retrievers in parallel, fetch top-K from each (e.g., 20), then fuse. Reciprocal Rank Fusion (RRF) is robust because it uses ranks not scores.
Practical tip: Keep the same embedding model for query and docs and version it; store model id with each vector.
2) Cross-encoder reranker
Why: Bi-encoder retrieval is fast but approximate. A cross-encoder reads the query and candidate jointly and produces much better relevance ordering — the usual production pattern is 100 → 20 → 5: fetch 100 candidates with a bi-encoder, run a lightweight reranker on top 20, and pass the top 3–5 chunks to the generator.
Cost control: Make reranking conditional. If the fused top score exceeds a confidence threshold, skip the cross-encoder.
Impact: Teams report 50–60% reductions in obvious hallucinations after adding reranking plus stricter prompting.
3) Prompt registry + versioning
Why: Prompts change often. Store canonical prompt templates, fallbacks, per-model variable bindings, and tag versions (v1, v2). Tie deployments to a prompt hash so rollbacks are deterministic.
How: Keep prompts in the repo or a small service (prompt-registry), include CI checks that run the eval harness against the new prompt before promotion.
4) Continuous RAGAS evals (Retrieval, Answers, Generation, Accuracy, Safety)
Why: You need measurable gates. Build a golden set (200–500 representative Q&A pairs) and automate daily smoke tests and weekly adversarial runs. Track precision@k, recall@k, faithfulness (does each claim cite a retrieved chunk?), and safety flags.
How: Run end-to-end CI checks on PRs that touch retrieval, chunking, prompts or model versions. Block deploys on regressions beyond a chosen threshold (e.g. >3% drop in recall@5).
5) Cost routing and caching
Why: Most inference cost is generation. Route cheap, high-recall queries to smaller models and reserve large models for long-context or high-confidence scoring. Cache reranker outputs and semantic-cache generated answers for repeated queries.
How: Use an intent/complexity classifier or cheap heuristics (query length, presence of numbers/IDs, metadata) to choose a model. Cache reranker scores keyed by (query_hash, candidate_set_hash) and only call cross-encoder when the cache misses or when confidence is low.
Two-sprint roadmap (minimal viable LLMOps)
Sprint 1 (1–2 sprints)
- Implement hybrid retrieval (dense embeddings + BM25).
- Add a bi-encoder retriever and expose top-20 candidate fetch.
- Basic telemetry: recall@5 and no-result rates.
Sprint 2 (1 sprint)
- Add cross-encoder reranker, gated behind a confidence threshold.
- Add a prompt registry with version tags.
- Wire up basic RAGAS smoke tests into CI.
Sprint 3 (optional)
- Productionize cost routing, semantic caching, and full eval infra with weekly adversarial runs.
Example: Python sketch for hybrid retrieve + conditional rerank
# simplified; replace with your DB/SDK calls
from sentence_transformers import SentenceTransformer, CrossEncoder
embedder = SentenceTransformer('all-mpnet-base-v2')
cross = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
def hybrid_candidates(query, vector_db, bm25_index, top_k=20):
q_emb = embedder.encode(query)
dense = vector_db.search(q_emb, k=top_k) # returns [(id, score), ...]
sparse = bm25_index.search(query, k=top_k) # returns [(id, score), ...]
# merge by rank using RRF
fused = reciprocal_rank_fusion(dense, sparse, k_const=60)
return fused[:top_k]
def conditional_rerank(query, candidates, rerank_threshold=0.35, rerank_top_n=5):
top_score = candidates[0].fused_score
if top_score > rerank_threshold:
return candidates[:rerank_top_n]
# build pairs and rerank
pairs = [(query, c.text) for c in candidates[:100]]
rerank_scores = cross.predict(pairs)
for c, s in zip(candidates[:100], rerank_scores):
c.score = s
candidates.sort(key=lambda c: c.score, reverse=True)
return candidates[:rerank_top_n]
This pattern keeps the reranker budgeted and predictable while improving top-K precision.
Observability, chunking and operational notes
- Chunking: use semantic/structure-aware chunking (paragraphs, headings). Parent-child chunking (small chunks for retrieval, larger parent for context) is often the highest-leverage change.
- Versioning: store embedding model id and index version alongside vectors. Treat embedding-model upgrades as a migration event.
- Tracing: instrument the full chain (query → embed → vectordb → rerank → LLM) with OpenTelemetry so you can attribute latencies and failures.
- CI: run RAGAS evaluations on PRs that touch retrieval or prompts. Block on regressions.
KPIs and gates to enforce before shipping
- Retrieval recall@5 ≥ target (domain-dependent, aim 85%+ for FAQs/internal docs).
- Faithfulness (RAGAS) below a tolerated hallucination rate (e.g., <5%).
- No-result rate below threshold; alert on spikes.
- Cost per query within budget after applying routing and caching.
Final takeaway
LLMOps for RAG systems isn’t an academic checklist — it’s engineering. Focus on deterministic pipelines, measurable gates (RAGAS), and a staged rollout: hybrid retrieval, reranking, prompt/version control, automated evals, and then cost routing. Do that and the model will behave.
Which one of the five would you tackle first on your codebase?
