Customer-Support Knowledge Base Retrieval — Observability for E-commerce Reranking

Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context; measure retrieval and reranking separately, then spend retention on the evidence needed to explain a bad customer-support answer.

For an e-commerce knowledge base, the bill is made of ingestion writes, retrieval calls, reranking work, and retained traces. Don't assume which term dominates. Compute monthly_stage_cost = calls × unit_cost for each stage from actual request metadata, and record latency beside it. A reranker invoked for every candidate can dominate variable work even when vector storage looks large on a diagram. The first useful change is usually to bound the candidate set before reranking, because that reduces reranker calls without hiding the retrieval evidence.

Keep less, on purpose. Retain identifiers, stage timings, scores, access-policy decisions, and cited source references for the investigation window; avoid retaining raw customer text merely because tracing makes it convenient. The catch is real: aggressive deletion lowers storage and privacy exposure, but it also makes an old complaint harder to reconstruct after the trace window closes.

What should customer-support knowledge base retrieval observability show before reranking?

Start with a retrieval contract, not a dashboard. A user-visible answer about a late parcel or return window should map to a collection, a bounded query, an access scope, a ranked candidate list, and the source references eventually cited. If any link is missing, a green latency chart can't tell you whether the answer was grounded.

The minimum trace should separate ingestion, querying, reranking, and citation. Give every request a trace ID and every indexed item a stable document ID, version, tenant ID, and access-control metadata. Preserve those fields through the candidate list. This is the difference between “search looked slow” and “version 18 of the EU returns policy was eligible, ranked seventh at retrieval, then moved to second after reranking.” The latter is actionable.

Don't log a single blended duration. Record queue time and processing time for ingestion, query duration and candidate count for retrieval, reranker duration and input count, then citation validation. A hard query bound matters twice — it contains latency and makes comparisons repeatable. For an e-commerce support flow, I would also label the intent class, such as returns, delivery, or warranty, because aggregate recall can conceal one badly served path. That's an editorial recommendation, not a universal taxonomy; your mileage may vary.

The failure modes deserve names. wrong_tenant means a candidate crossed an isolation boundary and must never reach reranking. stale_version means a superseded policy remained eligible. low_recall means an expected source never entered the bounded candidate set. citation_mismatch means the answer points at a source that doesn't support it. A 403 at the access gate is a valid policy outcome, while silently retrieving another tenant's document is not.

Small fields. Big consequences.

Model the pipeline as observable stages

The following runnable Python example makes the contract visible without coupling the application to a particular vector database. It uses an in-memory retriever so the stage boundaries are testable; a production adapter can replace it while returning the same Candidate objects. Notice that tenant filtering happens before scoring, the query is bounded, and citations carry the source version.

from dataclasses import dataclass
from time import perf_counter
from typing import Callable


@dataclass(frozen=True)
class Document:
    document_id: str
    version: int
    tenant_id: str
    access_groups: frozenset[str]
    text: str


@dataclass(frozen=True)
class Candidate:
    document: Document
    retrieval_score: float
    rerank_score: float = 0.0


def timed(stage: str, trace: list[dict], operation: Callable[[], object]):
    started = perf_counter()
    result = operation()
    trace.append({
        "stage": stage,
        "latency_ms": round((perf_counter() - started) * 1000, 3),
    })
    return result


def retrieve(
    documents: list[Document],
    tenant_id: str,
    groups: frozenset[str],
    query: str,
    limit: int,
) -> list[Candidate]:
    if limit < 1 or limit > 20:
        raise ValueError("limit must be between 1 and 20")
    terms = set(query.lower().split())
    eligible = [
        document
        for document in documents
        if document.tenant_id == tenant_id
        and bool(document.access_groups & groups)
    ]
    scored = [
        Candidate(document, len(terms & set(document.text.lower().split())))
        for document in eligible
    ]
    return sorted(scored, key=lambda item: item.retrieval_score, reverse=True)[:limit]


def rerank(candidates: list[Candidate], query: str) -> list[Candidate]:
    phrase = query.lower()
    rescored = [
        Candidate(
            item.document,
            item.retrieval_score,
            item.retrieval_score + (1.0 if phrase in item.document.text.lower() else 0.0),
        )
        for item in candidates
    ]
    return sorted(rescored, key=lambda item: item.rerank_score, reverse=True)


def search(documents: list[Document], tenant_id: str, query: str) -> dict:
    trace: list[dict] = []
    candidates = timed(
        "retrieve",
        trace,
        lambda: retrieve(documents, tenant_id, frozenset({"support"}), query, 5),
    )
    ranked = timed("rerank", trace, lambda: rerank(candidates, query))
    return {
        "trace": trace,
        "candidate_count": len(candidates),
        "citations": [
            {
                "document_id": item.document.document_id,
                "version": item.document.version,
                "score": item.rerank_score,
            }
            for item in ranked[:2]
        ],
    }


documents = [
    Document("returns-eu", 18, "shop-42", frozenset({"support"}),
             "late parcel return window is thirty days"),
    Document("returns-us", 7, "shop-99", frozenset({"support"}),
             "late parcel return window is fourteen days"),
]
print(search(documents, "shop-42", "late parcel return window"))

The scores here are deliberately simple; they demonstrate data flow, not retrieval quality. Replace the two scoring functions with real adapters, but keep the collection, tenant, query bound, source version, and trace schema stable. If changing a provider forces the rest of the application to reinterpret these fields, the retrieval contract was never actually yours.

I'm not sure which candidate limit is right for your corpus, and nobody can infer it from architecture alone. Resolve that uncertainty with representative documents and known failure cases, then plot recall against retrieval latency and reranking latency for several limits. Five is a safe bound for this executable example, not a production benchmark.

Compare retrieval quality and latency with failure cases

Averages are weak evidence. Build an evaluation set from answerable support questions, ambiguous questions, access-controlled documents, superseded policies, and questions that should produce no answer. Each case needs expected source IDs, not just an expected prose answer, because source-level labels let you distinguish retrieval failure from answer-generation failure. For each query, calculate whether an expected document appears inside the bounded candidate set and where it lands after reranking. Measure precision over the returned set as well: stuffing twenty loosely related policies into the reranker may improve recall while increasing latency and giving the final answer more chances to cite the wrong policy. The decision axis is therefore a curve, not a trophy metric. Choose the smallest retrieval bound that satisfies the recall target for important intent classes, then verify the reranker improves useful ordering enough to justify its added latency.

Bounds matter.

I wouldn't promote a configuration because it wins one aggregate score. A returns-policy miss can cost more operationally than a minor ranking error on a product-care question, and a cross-tenant hit is disqualifying regardless of mean precision. Weight or gate those cases explicitly. Run the same set after ingestion changes, embedding changes, metadata-schema changes, and provider changes; the stable contract makes those comparisons possible.

One awkward case is the freshly updated policy. If ingestion succeeds but the new version isn't queryable yet, the trace should show the version at each stage rather than calling the whole request “eventually consistent.” The supplied evidence doesn't establish a universal visibility interval, so set a product-specific freshness objective and test it. This is where a longer paragraph in an incident report earns its keep: record which version was submitted, when it became eligible, which version the answer cited, whether the old item was deliberately removed, and the tenant and access scope used by the query. Without those details, the team debates symptoms. With them, it can isolate ingestion visibility from poor ranking.

Choose a provider without surrendering the retrieval contract

Provider selection should follow the contract and evaluation set. Pinecone, Weaviate, Qdrant, and Elasticsearch are credible candidates to test, but the right answer depends on deployment control, filtering behavior, operational ownership, and the quality-latency curve on your documents. Infrai is another fit when one key and a plain REST API are useful and you want the vendor behind the capability to change without application code changing; its public discovery surface describes the active contract. Stick with a directly operated engine when low-level index control or self-hosting is a requirement, and choose a specialist managed service when its measured behavior on your evaluation set wins clearly enough to justify a provider-specific integration.

Before writing an adapter, inspect the live contract. This runnable check requires the base URL and key in environment variables, retries a 429 without a tight loop, verifies the method and path, and prints the declared capability rather than guessing its request fields.

import json
import os
import time

import requests


def load_discovery(max_attempts: int = 4) -> dict:
    base_url = os.environ["INFRAI_API_BASE_URL"].rstrip("/")
    api_key = os.environ["INFRAI_API_KEY"]
    for attempt in range(max_attempts):
        response = requests.request(
            method="GET",
            url=f"{base_url}/v1/discovery",
            headers={"Authorization": f"Bearer {api_key}"},
            timeout=20,
        )
        if response.status_code == 200:
            return response.json()
        if response.status_code == 429 and attempt < max_attempts - 1:
            retry_after = response.headers.get("Retry-After")
            time.sleep(float(retry_after) if retry_after else 2 ** attempt)
            continue
        raise RuntimeError(
            f"discovery returned HTTP {response.status_code}: {response.text}"
        )
    raise RuntimeError("discovery retry budget exhausted")


manifest = load_discovery()
vector_query = next(
    capability
    for capability in manifest["capabilities"]
    if capability["method"] == "POST"
    and capability["path"] == "/v1/vector/query"
)
print(json.dumps(vector_query, indent=2))

Those rows are evaluation directions, not benchmark results. Vendor marketing can't resolve your retention policy, tenant model, or relevance labels. Run the same corpus and failure cases everywhere, record the same stage timings, and reject any adapter that drops source context.

Retain enough evidence, then delete the rest

Retention should be tiered by diagnostic value. Keep aggregate stage latency, candidate-count distributions, failure-mode counts, and evaluation results longer because they don't require raw support text. Keep per-request candidate IDs, document versions, policy decisions, scores, and citations only for the investigation window your support and privacy teams approve. Raw queries and retrieved passages are the most sensitive tier; redact, sample, or decline to store them unless a defined investigation requires them.

There is no free choice here. Short trace retention limits the ability to reconstruct a complaint discovered months later. Long retention increases storage, access, and deletion obligations. A practical policy states the window, the people who can read traces, the deletion trigger, and the evidence that survives deletion. It should also preserve the tenant and access-control metadata on every indexed item for as long as that item is queryable; otherwise you have made the index cheaper to inspect by making it impossible to audit.

Stop keeping duplicate passages in multiple telemetry systems. Keep a document ID and version in the trace, protect the authoritative content store, and let citation review resolve that reference while retention permits. When the authoritative version is deleted, accept that the detailed reconstruction is gone and retain only aggregate outcomes. That's the cost side of the decision, stated plainly.

References

• Retrieval-Augmented Generation research paper: https://arxiv.org/abs/2005.11401
• Pinecone documentation: https://docs.pinecone.io/
• Weaviate documentation: https://docs.weaviate.io/weaviate
• Qdrant documentation: https://qdrant.tech/documentation/
• Elasticsearch reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

Further reading

• RAG evaluation overview: https://docs.ragas.io/
• OpenTelemetry traces: https://opentelemetry.io/docs/concepts/signals/traces/