Pick Top K and Similarity Thresholds for Production Vector Queries (With Freshness SLOs)

The page says a production vector query over clinical PDFs is returning too many unsupported answers. To pick top K and a similarity threshold, on-call needs evidence recall and source freshness, not a score copied from another corpus. The trace shows why: several weakly related passages reached the generator, while the passage containing revised guidance was absent because its source version had not finished indexing.

TL;DR: pick top K and the similarity threshold together from a labeled query set, then gate them with two separate production signals: retrieval quality and source freshness. Start with a permissive candidate K, reject candidates below a calibrated threshold, and cap final context by document and token budget. The useful setting is the smallest context that meets a recall target for current documents without breaching latency and false-positive budgets.

A high K can hide poor chunking by filling the prompt with neighboring text. A strict threshold can make newly worded material disappear. The page should fire on the failure users experience, but an earlier warning should tell on-call whether retrieval quality, ingestion freshness, or both are deteriorating.

What should have paged before unsupported answers appeared?

The late signal is an answer-level groundedness failure. It matters, but it arrives after retrieval and generation have spent their budgets. Earlier signals live at the retrieval boundary. For a fixed evaluation set, record whether at least one accepted chunk contains the labeled evidence, how many accepted chunks are irrelevant, and whether the newest eligible source version was searchable.

Call the first signal evidence recall. Call the second noise acceptance. Freshness stays separate: searchable_at - source_observed_at measures indexing lag, while a source-version mismatch shows that a query used an older revision. One aggregate success rate smears those failures together.

The trace should carry the retrieval decision, not PDF text: query class, embedding configuration identifier, candidate K, accepted count, threshold version, source version IDs, chunking-policy version, and timings. Clinical text can contain sensitive information, so raw questions and passages need their own access, retention, and redaction policy; they do not belong in routine metric labels.

Short traces beat blind dashboards.

Page when the user-facing unsupported-answer SLO burns too quickly. Warn when labeled evidence recall falls below its objective for more than one evaluation window, or when indexing lag consumes the freshness budget. Treat noise acceptance and context size as investigation signals rather than declaring every fluctuation an incident.

The exact objectives must come from the service's risk review and observed baseline. Inventing a percentage would be false precision. Each objective needs a measurement population, window, owner, and response.

How should production vector queries pick K and a similarity threshold?

Top K controls how many nearest candidates the search stage may inspect. The similarity threshold controls which scored candidates may enter context. Raising K creates more opportunity to recover evidence and to admit noise; raising the threshold removes weak matches and can also remove valid evidence phrased differently from the question.

Build a labeled set from the actual job: questions over current clinical PDFs, with source version and evidence span recorded. Include abbreviation-heavy questions, copied phrases, paraphrases, questions whose answer is absent, and questions that become answerable only after a revised PDF arrives. Split tuning cases from holdout cases, or the selected pair will merely memorize calibration data.

For every candidate pair, replay the same questions and record evidence recall, irrelevant chunks accepted, empty retrievals, context tokens, and latency. Evaluate by query class and document version, not just one average. A setting that looks acceptable overall can fail revised documents after headings or tables change.

Scores are not probabilities. Their scale depends on the embedding model, vector normalization, distance function, corpus, and query distribution. A threshold of 0.8 has no portable meaning without those details. Store threshold configuration beside embedding and chunking versions, then recalibrate after changing any of them.

No shortcut survives those changes.

This generic Go function assumes the search adapter converted its native distance into a documented higher-is-better score. It also limits repeated passages from one document so a long PDF cannot crowd out every other source.

package retrieval

import "sort"

type Candidate struct {
    DocumentID string
    VersionID  string
    ChunkID    string
    Score      float64
    Tokens     int
}

type Policy struct {
    CandidateK     int
    MinScore       float64
    MaxContext     int
    MaxPerDocument int
}

func Select(candidates []Candidate, p Policy) []Candidate {
    if p.CandidateK <= 0 || p.MaxContext <= 0 || p.MaxPerDocument <= 0 {
        return nil
    }
    sort.SliceStable(candidates, func(i, j int) bool {
        return candidates[i].Score > candidates[j].Score
    })
    limit := p.CandidateK
    if len(candidates) < limit {
        limit = len(candidates)
    }

    selected := make([]Candidate, 0, limit)
    perDocument := make(map[string]int)
    usedTokens := 0
    for _, c := range candidates[:limit] {
        if c.Score < p.MinScore || c.Tokens <= 0 {
            continue
        }
        if perDocument[c.DocumentID] >= p.MaxPerDocument {
            continue
        }
        if usedTokens+c.Tokens > p.MaxContext {
            continue
        }
        selected = append(selected, c)
        perDocument[c.DocumentID]++
        usedTokens += c.Tokens
    }
    return selected
}

Production complexity belongs around this function: version validation, per-query-class policies, deduplication, reranking, access control, and refusal when no acceptable evidence remains. Keep those choices observable rather than hiding them in an opaque search call.

Chunking and freshness change the apparent optimum

K cannot rescue arbitrary chunks. If a parser joins an entire section, its embedding may blur the sentence a question needs; if it splits every line, retrieval may find a fragment without its qualifying sentence or table header. Evaluate chunking with evidence containment and context sufficiency. Use overlap where a boundary would otherwise separate a claim from its qualifier.

Tables need special attention. Preserve headers with rows, page and section provenance, and a stable link to the source version. A retrieved number without its unit, population, or revision is inadequate evidence, even with an excellent vector score.

Freshness can masquerade as threshold trouble. Suppose a revised PDF uses new terminology. If its old version remains searchable while the new one is still being chunked, lowering the threshold may retrieve more stale passages without making the answer current. Prepare chunks under the new source version, verify the inventory, make that version eligible atomically, and retire the previous version according to policy. Queries should record which version was eligible at decision time.

This creates a capacity-planning requirement. Size ingestion for bursts, not average document volume. Ask whether a revision burst can be parsed, embedded, indexed, and made searchable inside the freshness objective while query traffic continues. Track work by pages or chunks as well as documents because PDF sizes vary. Reserve headroom from measured service time and arrival patterns; no universal multiplier is honest.

Instrument the decision, then deploy it gradually

Emit counters and histograms for candidate count, accepted count, empty results, accepted score distribution, context tokens, retrieval latency, and source-version age. Keep document and query identifiers in protected traces or logs, not metric dimensions.

Offline replay comes first. Shadow evaluation comes next: compute the proposed policy beside the current policy without changing visible answers, compare selected chunk IDs and labeled outcomes, then expose the new policy to a controlled traffic slice. Roll back on SLO impact. A configuration version in every trace removes guesswork about which threshold served a request.

The buy-versus-build decision is less about nearest-neighbor syntax than operational ownership:

A managed component can reduce maintenance while adding dependency and migration constraints. A self-hosted component exposes every control while assigning every failure mode to the platform team. Neither choice fixes unlabeled data, stale versions, or careless chunk boundaries.

This method has a limitation: it is not suitable when the team cannot label evidence or determine which document version was eligible, because recall then has no defensible denominator. In that case, the practical trade-off is to delay automated answers, provide conventional keyword filtering with direct document links, or keep a human review step until labels and version lineage exist. The same caution applies to tiny or rapidly shifting query samples; a finely tuned threshold can look precise while representing almost none of the production distribution.

Precision can be theater.

The threshold has its own false-positive cost

Return to the page. Once freshness and retrieval quality are separate, on-call can see that the answer failure followed source-version lag, not a sudden collapse in vector similarity. Restore the freshness objective and replay affected evaluation questions. Changing the threshold during that incident alters a second variable and may admit more unrelated text.

Then tune with evidence. Choose the smallest accepted context that meets the evidence-recall objective on holdout data, stays within latency and token budgets, and preserves an explicit no-evidence outcome. Revisit it when the embedding configuration, chunker, corpus mix, or question distribution changes.

An overly sensitive alert has a cost. If every small score shift pages on-call, responders learn that the signal rarely maps to user impact. If paging is too lax, users find the regression first. Put user-facing SLO burn on the pager; put leading indicators on warnings with documented escalation criteria. The retrieval threshold decides what text enters an answer. The alert threshold decides when a human loses sleep.

Further reading

References:

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401