Two‑Stage Retrieval & Reranking: A Technical Guide

Recall first, precision second. This guide explains how to separate candidate generation from relevance refinement, how to tune the funnel, and which metrics keep the system both fast and trustworthy.

📋 Concise Diagnosis Table

🧭 Decision Flow

flowchart TD
    A["Start: User Query"]
    B["Dense / BM25 Search → Top‑N"]
    C["Apply Filters (date, jurisdiction)"]
    D["Reranker LLM scores candidates"]
    E["Select Top‑k"]
    F["Assemble Prompt for Generator"]
    G["Generate Answer"]
    H["Return to User"]
    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H

🏗️ Architecture & Responsibility Matrix

*Latency targets are illustrative; actual Service‑Level Agreements must be defined per deployment.

🐍 Teaching‑Only Python Helper

# teaching-only: helper to decide top‑N / top‑k based on policy constants
import os
from typing import Tuple

# ---- Configuration (replace with env‑vars or config files in production) ----
MAX_PROMPT_TOKENS = 1024          # hard limit for downstream generator
MAX_LATENCY_MS    = 1500          # end‑to‑end budget
DEFAULT_TOP_N    = 200
DEFAULT_TOP_K    = 5

def decide_funnel_sizes(
    estimated_index_tokens: int,
    estimated_rerank_latency_ms: int,
) -> Tuple[int, int]:
    """
    Return (top_n, top_k) that respects MAX_PROMPT_TOKENS and MAX_LATENCY_MS.
    Assertions guarantee deterministic behaviour for unit testing.
    """
    # 1️⃣ Start with defaults
    top_n = DEFAULT_TOP_N
    top_k = DEFAULT_TOP_K

    # 2️⃣ Adjust top_n if the token budget would be exceeded even before reranking
    if estimated_index_tokens > MAX_PROMPT_TOKENS:
        # Reduce the pool size proportionally, but never below 1
        top_n = max(1, top_n * MAX_PROMPT_TOKENS // estimated_index_tokens)

    # 3️⃣ Adjust top_k to keep the final prompt under the token limit
    #    Assume an average snippet length of 200 tokens (empirical, not arbitrary)
    avg_snippet_len = 200
    max_k = MAX_PROMPT_TOKENS // avg_snippet_len
    top_k = min(top_k, max_k)

    # 4️⃣ Ensure the overall latency budget is not violated
    #    +500 ms is a placeholder for generator inference; replace with real estimate in production
    total_latency = estimated_rerank_latency_ms + 500
    assert total_latency <= MAX_LATENCY_MS, (
        f"Latency {total_latency} ms exceeds budget {MAX_LATENCY_MS} ms"
    )

    # 5️⃣ Return a deterministic tuple
    return top_n, top_k

# Example (for notebooks or documentation):
# top_n, top_k = decide_funnel_sizes(estimated_index_tokens=1200,
#                                   estimated_rerank_latency_ms=250)
# print(top_n, top_k)

📜 Architectural Decision Records (ADRs)

ADR 1 – Choosing a Dense Vector Index

ADR 2 – Reranker Scoring Threshold

✅ Evaluation & Maintenance Checklist

• Recall Test: Run a quarterly BM25 + dense recall benchmark on a labelled query set (≥ 200 samples). Record the percentage of known relevant passages retrieved in Top‑N.
• Precision Test: After reranking, compute Precision@k (k = 5, 10) on the same set. Verify against the targets in ADR 2.
• Latency SLA: Measure end‑to‑end latency (index → rerank → generator) under a representative load (≥ 100 QPS). Alert if the 95th‑percentile exceeds MAX_LATENCY_MS.
• Token Budget Audits: Log prompt_tokens_before and prompt_tokens_after. Ensure average reduction meets the > 20 % target.
• Compliance Filter Validation: Unit‑test all deterministic filter rules (date, jurisdiction, confidentiality) with a deterministic test harness; any failure must block deployment.
• Model Drift Monitoring: Track hallucination rate using a synthetic fact‑checking prompt. If the rate rises above 5 %, trigger a prompt‑engineering review (do not automatically fine‑tune).
• Documentation Sync: Verify that any change to MAX_PROMPT_TOKENS, MAX_LATENCY_MS, or rerank.threshold is reflected in this guide and the corresponding ADRs.
• Backup & Rollback: Archive the exact vector index snapshot and reranker prompt version used for each production release. Enable deterministic rollback if a new release violates any acceptance criterion.

📚 References

• Microsoft RAG concepts – Azure Foundry documentation: https://learn.microsoft.com/en-us/azure/foundry/concepts/retrieval-augmented-generation
• LoRA: Low‑Rank Adaptation of Large Language Models – research paper: https://arxiv.org/abs/2106.09685

(The LoRA paper is cited for understanding lightweight adaptation; it is not the original RAG publication.)

Editorial Note

• A symptom listed in the diagnosis table is a hypothesis, not proof of the root cause.
• RAG does not guarantee factual accuracy, instant compliance, or complete coverage; deterministic business rules and live‑value APIs must be used where up‑to‑date facts are required.
• LoRA updates freeze the base model weights and add low‑rank matrices; they do not inherently preserve all original behaviours—regression testing remains essential.
• When a request requires a live value (e.g., a current market price), retrieve it from an authoritative API rather than a cached document.

End of guide.