Inside VOICE-RAG: From Speech to Grounded Answers

Pavan Ganesh Indupudi (Infernix) · Originally published at the-infernix.github.io

Ask most RAG demos "Who is the Prime Minister of India?" and they will happily hallucinate an answer from whatever passage happens to look semantically close. VOICE-RAG — built for HH Goa 2026 — answers that question with a refusal and a number:

Refused: Top retrieval score 0.779 below relevance floor 0.840

That sentence is the whole thesis. The system answers spoken questions in English, Hindi, Gujarati, and Telugu — strictly from a curated 40,000-passage corpus — and refuses when the corpus does not contain the answer. This article covers why that turned out to be the hard part, and how the 56 ms core around it was built.

The problem: fluent answers with no evidence

Retrieval-augmented generation fails in two directions. Too loose, and the generator papers over missing evidence with fluent nonsense. Too strict, and the system refuses questions it could have answered. Most demos only tune for the first failure — making answers sound grounded. We optimized for the second failure being visible: every refusal states its score and its floor, and the UI shows the entire evidence chain either way.

Constraints

• Voice is the primary input. Hold-to-talk capture, 16 kHz mono WAV, Sarvam saaras:v3 speech-to-text. Voice adds roughly 1 s of STT; everything after it had to be fast enough to disappear.
• Four languages, one index. English, Hindi, Gujarati, Telugu over the MSMARCO-XI corpus — 10k passages per language, 384-dim multilingual E5 embeddings.
• Corpus-bounded by design. World-knowledge questions with zero supporting passages must be refused, not answered from parametric memory.
• No bundler. The frontend is a vanilla-JS SPA served directly by FastAPI. Zero node_modules, zero build step.

Architecture: three guards around a 56 ms core

The pipeline is a strict sequence, and every stage reports its latency and its decision into a trace the UI renders as a "why this answer" chain:

The evidence drawer and latency trace — every guard decision is user-visible, not server-side secret sauce.

Input guard → embed → vector search → relevance guard → generate → grounding guard. Measured medians (n = 2,000 benchmark queries):

Design decision
Default to extractive answering — pick the best-supported sentence from the top passage — and make LLM generation strictly opt-in. Extractive answers are 0.09 ms and grounded by construction; generative answers take multiple seconds through OpenRouter and need a citation contract to stay honest. Latency budget first, generative flair when asked for.

The refusal engine: floors calibrated from measurements

The relevance guard is one comparison — top-1 cosine score against a per-language floor — but the floors are not guesses. They were calibrated by measuring score distributions: known-relevant queries clustered around P25–P50 ≈ 0.84–0.86, off-corpus junk around ≈ 0.79–0.85, and the floors were set at the separation point:

The query's language selects the floor — not the top chunk's — so cross-language junk cannot sneak through under a borrowed threshold. Language detection is Unicode-script based (Telugu block → te, Gujarati → gu, Devanagari → hi, else en), and auto-detected language applies a soft +0.03 rerank bonus toward same-language passages over a 3× candidate pool.

Q: Who is the Prime Minister of India?   → REFUSED  (0.779 < 0.840)
Q: Who won the FIFA World Cup 2022?      → REFUSED  (0.796 < 0.840)
Q: How much does an Xbox 360 cost?       → answered ($80–$250, conf 0.933)

The grounding guard: answers must be traceable

Passing the relevance gate is not enough. The grounding guard verifies that the answer is actually supported by retrieved text: token overlap ≥ 0.231 passes instantly; below that, an embedding-similarity check (≥ 0.794) runs as fallback. Ungrounded generations are replaced with the extractive answer or refused outright.

When generative mode is enabled, the LLM must return strict JSON — {answer, citations[], confidence} — citing only supplied passages. Malformed output gets exactly one retry. Invented or out-of-range citations are dropped, and if none survive, the pipeline falls back to extractive. An ungrounded generation can never reach the user.

Experiments: 2,000 queries, and the hybrid detour

The benchmark suite runs 2,000 corpus queries (500 per language, balanced, seed-fixed) against the prebuilt index:

Guard activity in that run: 1,893 answered · 107 refused — weak-match queries correctly declined rather than answered with junk. The single P100 outlier (273 ms, Gujarati) was one OS-scheduling event; the next-worst query across all languages was 178 ms.

We also built and measured a hybrid BM25 + dense leg (bm25s, reciprocal rank fusion), evaluated against MSMARCO's own gold labels:

Trade-off
Hybrid bought +0.4 pp hit rate for ~1 ms and a large maintenance surface. Measured parity means dense-only stays the default; the sparse leg ships flag-ready behind retrieval.hybrid.enabled and off by default. RRF is used for candidate-pool expansion only — final ordering is always cosine, which keeps retrieval scores calibrated with the refusal floors. A hybrid ordering would have silently invalidated every floor we measured.

Every search also runs against a 200 ms deadline; skipped legs surface as explicit degradation strings (sparse_leg_skipped_deadline) in the stage trace, never as silent timeouts.

Limitations

• Corpus-bounded knowledge. "What is the capital of India?" has zero supporting passages and is correctly refused — a feature, but it surprises users expecting a chatbot.
• The borderline zone is real. A band around 0.84–0.86 overlaps between weak-relevant and strong-junk retrievals. Per-language floors trade a little recall for a lot of precision; some answerable questions get refused.
• Voice adds ~1 s. Sarvam STT dominates end-to-end time; the RAG core's 56 ms is invisible under it.
• Generative mode is opt-in precisely because it breaks the latency budget.

Lesson
Calibrate thresholds from measured score distributions, not intuition. The floors took an afternoon of histogram work and saved the product: without them, the system would have been confidently wrong in exactly the situations where being wrong matters most. A refusal with a stated score is also infinitely more debuggable than a silent abstention.

Reproducibility

The prebuilt index (40k × 384-dim vectors) ships with the repo, so first run needs no indexing step. Raw per-query benchmark data is committed at bench/results/results.json.

git clone https://github.com/The-Infernix/VOICE-RAG.git
cd VOICE-RAG
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn api.main:app --port 7860   # SPA at http://localhost:7860

pytest tests/ -q                   # 178 passing
python bench/run_bench.py --limit 2000      # latency
python bench/eval_retrieval.py               # HitRate@10 / MRR@10 vs gold
python bench/run_bench.py --limit 500 --hybrid --soft-lang \
  --out bench/results/ab_hybrid              # the hybrid A/B above

Takeaways

1. Refusal is a product feature, not an error path. Make it measurable, calibrated, and visible.
2. Guard every stage independently — input, relevance, grounding — and surface every decision in the UI trace.
3. When a measured improvement is inside the noise, ship the simpler system and keep the complex one behind a flag.
4. Keep retrieval ordering aligned with whatever your thresholds were calibrated against.

References

• VOICE-RAG repository — bench results, 178-test suite, config
• bench/results/summary.json — the latency numbers quoted above
• bench/eval_retrieval.py — gold-label evaluation against MSMARCO is_selected labels
• guardrails/relevance_guard.py — the calibrated per-language floors
• Built for HH Goa 2026, Task 2

Pavan Ganesh Indupudi (Infernix) — CS student at Andhra University building privacy-preserving AI. Full article and more at the-infernix.github.io · GitHub · LinkedIn