How my agent actually finds answers — and what happens when it doesn't
Part 5 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The companion repo contains the full code.
"What's your refund policy?"
Something has to know the answer. The model doesn't. Not reliably.
The answer lives in documents the company wrote. Getting the right one in front of the model at the right moment has an intimidating name: retrieval-augmented generation (RAG). And most explanations make it sound like magic.
It's a pipeline. Score the documents, rank them, hand back the best few. That's all. The interesting part is what you do with the score.
Retrieval is a tool, not context stuffing
Fuzzy results behind a hard contract — that's the split this system is built on, and here it is made real.
The agent doesn't get knowledge silently injected into its prompt. It gets a tool, the same way it gets customer lookup:
// dev/tonal/support/knowledge/KnowledgeBase.java
public interface KnowledgeBase {
/** Returns up to query.topK() articles, best match first. */
List<ScoredArticle> search(Query query);
}
The agent decides when to search and what to ask. It never redefines what searching means, and every call is visible: query in, ranked articles with scores out.
flowchart LR
A["Agent needs an answer"] --> B["Query: text + topK"]
B --> C{"Scorer"}
C --> D["Ranked articles + scores"]
D --> E["Top-k back to the agent<br/>as tool result"]
C -.-> F["keyword overlap (shipped)"]
C -.-> G["embeddings (same port)"]
classDef step fill:#eef2f6,stroke:#8fa3b8,color:#24313f
classDef decision fill:#f7f4ec,stroke:#b3a988,color:#24313f
classDef alt fill:#f7f9fb,stroke:#c5d1dc,color:#24313f
class A,B,D,E step
class C decision
class F,G altDeterministic first, semantic later
Here's the part that breaks with convention: the shipped implementation scores articles by keyword overlap — plain code, no embeddings, no API key.
// dev/tonal/support/knowledge/KeywordScoringKnowledgeBase.java
public List<ScoredArticle> search(Query query) {
Set<String> queryTokens = tokens(query.text());
return articles.stream()
.map(article -> new ScoredArticle(article, score(article, queryTokens)))
.filter(scored -> scored.score() > 0)
.sorted(Comparator.comparingDouble(ScoredArticle::score).reversed())
.limit(query.topK())
.toList();
}
Why ship the dumb version? Because it's fully assertable.
Four tests pin the whole behaviour:
- the right article ranks first for a policy question
- topK actually limits results, zero token overlap returns empty (not "closest guess")
- ordering is strictly by score.
When an embedding-backed scorer replaces this class — same port, better matching on paraphrases — those tests define what honouring the contract means. Swap the implementation, keep the guarantees.
Scores are also why retrieval is debuggable. Every match carries its number:
$ java ... dev.tonal.support.knowledge.KnowledgeMain
# GET http://localhost:8080/rag/search?q=refund&k=3
[1.00] Refund Policy (billing)
# GET http://localhost:8080/rag/search?q=xylophone
No articles matched.
When the agent later cites a policy, you can replay the exact query and see exactly what it was shown. No black box between the corpus and the answer.
What wrong looks like
Retrieval being probabilistic means sometimes the ranker surfaces the wrong document — a rate-limit page for an SLA question. That's a failure mode like any other in this system: enumerated, mitigated, measured.
The mitigation starts with honesty about scores (a 0.2 match should be treated differently from a 1.0), continues through grounding answers in what was actually retrieved rather than what the model remembers, and ends with the eval suite scoring whether answers follow from sources. A wrong document isn't a bug you fix once. It's a quality property you track.
The pattern generalizes past support bots:
- enterprise search ranks but humans decide what's authoritative;
- clinical guideline systems surface candidates while physicians own the prescription;
- legal research tools find precedents while counsel argues them.
Ranked candidates plus human-or-rule judgment beats either pure search or pure generation everywhere it matters.
