Vector Search Latency Spikes: Isolate Top K and Payload Amplification Under Load

Keep the vector index narrow, cap the candidate count, and fetch document bodies only after ranking. That is the first decision to test when vector search latency spikes under load: increasing top k and returning large payloads can turn a tolerable nearest-neighbor query into expensive materialization and network work, even when the index lookup itself stays steady.

Short answer: measure the lookup, payload fetch, serialization, and transfer separately at the same concurrency and filter selectivity as production. Change one knob at a time. A lower top k is not a fix if it drops the evidence the reader needs.

Consider a fintech help center that also watches policy pages for changes and alerts on the diff. Each revision may create new searchable passages, but an alert needs a stable page identity and revision reference, not a second full copy of the page embedded in every search hit. The primary constraint here is index cost at scale; latency is the symptom that exposes a poor boundary between vectors, metadata, and document bodies.

What are the decision's invariants?

Record three invariants before tuning. A search hit must identify an immutable content revision so the text shown to a user matches the text used for retrieval. A page-change alert must compare committed revisions, rather than treating a transient search result as the source of truth. Finally, the retrieval budget must be measured at the requested concurrency; a fast median at idle cannot establish behavior under load.

There are two failure boundaries. First, top k is a candidate count, not a guarantee of k useful answers: filtering, duplicate passages, and thresholding can leave fewer usable results. Second, a vector hit's metadata may fit comfortably in memory while its associated body does not. For example, 100 hits carrying 8 KiB each imply about 800 KiB of body data before envelope overhead, versus about 40 KiB for five such hits. These are arithmetic illustrations, not measurements or recommended limits. Revisions multiply stored passages too, so the cost of keeping every body in the search index persists between queries. If the same policy page has several indexed revisions, returning the latest page body for an older hit breaks the citation even when the prose happens to be similar; a revision identifier must remain part of the lookup key, and retention rules must not delete that revision while an alert can still reference it.

That mismatch is subtle.

The middle option is the decision for this workflow. It separates the change detector's durable record from the search index's replaceable projection. That distinction matters when a policy page is edited twice while an alert is queued: the diff and the cited search excerpt must each resolve to the revision they name, even if a newer revision has already been indexed.

How do you debug vector search latency spikes under load?

Instrument four intervals: query admission to index response, index response to selected revision fetch, fetch to serialized response, and response transfer. Capture requested top k, returned count, payload bytes, filter selectivity, concurrency, and revision-fetch misses with each trace. Compare p50 and p95 for the same workload shape; a pooled p95 can mix small help-center questions with wide policy-change investigations and conceal the cause. Queue time belongs in the trace as well. Otherwise, a saturation-induced wait looks like a slow nearest-neighbor calculation.

Start with a small experiment matrix rather than a single before-and-after run: hold query vectors and filter predicates fixed, vary top k, then vary whether bodies are returned, while keeping offered concurrency constant. Watch both latency and the fraction of questions for which the selected passages actually contain the needed policy clause. If p95 grows when top k increases only when bodies are included, inspect payload fetch and encoding before changing index parameters. If it grows even for identifier-only results, investigate index traversal, filters, memory pressure, and queueing. A changed corpus invalidates the comparison; freeze a revision snapshot for the test.

This is also a consistency test. An index is a derived view; a newly committed page revision may not yet appear in a query. The alerting path should take its diff from the revision store and use the search index only to find relevant passages, with an explicit policy for indexing lag. Never infer that an absent search hit means no page changed.

What belongs on the critical path?

The following Python sketch makes the boundary explicit. The interfaces are application contracts, not claims about a particular database; search_ids returns only identifiers and scores, while get_revision retrieves an exact immutable revision. The bounded fetch is intentionally after ranking.

from dataclasses import dataclass


@dataclass(frozen=True)
class Hit:
    page_id: str
    revision_id: str
    passage_id: str
    score: float


def retrieve(question_vector, index, revisions, candidate_k=30, context_k=5):
    if not 1 <= context_k <= candidate_k <= 100:
        raise ValueError("invalid retrieval bounds")

    hits = index.search_ids(question_vector, limit=candidate_k)
    selected = []
    seen = set()
    for hit in hits:
        key = (hit.page_id, hit.revision_id, hit.passage_id)
        if key in seen:
            continue
        seen.add(key)
        passage = revisions.get_revision(key)
        if passage is None:
            raise LookupError(f"missing indexed revision: {key}")
        selected.append((hit, passage))
        if len(selected) == context_k:
            break
    return selected

The numbers in that example are illustrative configuration, not universal limits. Set them using a labeled question set and a measured response budget. In a real deployment, bound the total bytes as well as the count, cap concurrent revision fetches, and distinguish a missing revision from an ordinary no-results response. A retry of a permanently missing revision only adds load. Preserve the revision identifier in logs and citations so an operator can correlate a search result with the page diff that generated the alert.

Roll out the split path behind a controlled traffic percentage and compare latency distributions, response bytes, revision-fetch errors, and retrieval coverage against the existing path. The operational trade-off is real: two stores and a join need clear ownership, deployment ordering, and an on-call procedure for index lag. A cheap index that repeatedly fails to resolve selected passages is not a successful optimization.

When is the rejected design reasonable?

Keeping full bodies in the search index can be reasonable for a small, bounded corpus with short passages, low concurrency, and no requirement to resolve historic revisions. It removes the second lookup. Reject it here because versioned policy pages grow over time and the page watcher requires durable, exact revision identity; repeatedly carrying old bodies through the search hot path couples alert history to query payload size.

Treat the index as a locator and measure the rest of the path. If identifier-only searches still spike at fixed concurrency, this decision has narrowed the investigation, not solved it: examine admission queues, filter selectivity, and index memory behavior with the same frozen query set. The acceptance test is stable tail latency without losing the passage needed to explain a change, and exact revision resolution when an alert is inspected later.

References

• https://arxiv.org/abs/2005.11401
• https://www.w3.org/TR/trace-context/
• https://opentelemetry.io/docs/specs/semconv/http/http-spans/