Python Retrieval Architecture — Evaluation Sets for a Restaurant Menu Assistant

Short answer: build the restaurant menu assistant around a versioned evaluation set before choosing a retrieval method. Measure whether the retriever finds the right menu fact, rejects unavailable items, and prefers the newest applicable record; then retain only the artifacts needed to reproduce a failed decision.

The bill is made of menu ingestion, index storage, retrieval work, generation, evaluation runs, and retained traces. The dominant term cannot be declared from an architecture diagram. Measure it as daily cost = changed items processed + queries served + evaluations run + artifacts retained, using the units charged or consumed by the actual stack. For a menu that changes in small batches, reprocessing every historical description can dominate ingestion; for a busy assistant, repeated retrieval and generation can dominate instead. The first useful optimization is therefore not a smaller model or a fashionable index. It is separating changed menu versions from unchanged ones and running the smallest evaluation slice that still covers the affected restaurant, service window, dietary claim, and availability state.

Keep the current source snapshot, its content hash, the evaluation case version, retrieved identifiers and scores, and the final pass/fail result. Stop keeping duplicate chunks and full successful traces after the audit window your team has actually approved. The catch is forensic depth: if a regression appears after those artifacts expire, the team may be able to reproduce the current answer but not the exact old ranking.

What actually drives retrieval cost and retention?

A useful cost model separates work caused by a menu change from work caused by a diner question. That distinction matters because freshness and chunking pull in opposite directions: smaller chunks reduce the text rewritten by a price or availability edit, while excessively small chunks can detach an item from its section, restaurant, modifiers, or service window. Larger chunks preserve context, but one small edit invalidates more indexed material and can make a retrieved passage contain conflicting facts.

Do the arithmetic with observed counts. If 40 items change, the affected evaluation slice should be derived from those 40 item IDs and their dependencies, not from an invented percentage target. A nightly full run still has value as a backstop, but its cadence should follow the time in which a wrong allergen, price, or availability answer remains operationally tolerable. I'm not sure there is a universal cadence; menu publishing latency and the team's correction process resolve that question.

Small records win here.

How should evaluation sets shape retrieval architecture for a restaurant menu assistant?

An evaluation case is a contract, not a bag of representative questions. Each case needs a query, the menu version that was valid at evaluation time, acceptable source item IDs, explicitly forbidden IDs when a plausible stale or cross-restaurant result exists, and facets that make failures diagnosable. Those facets should include restaurant, locale, service window, dietary intent, availability state, and question type when they affect the expected answer.

The crucial split is between retrieval and answering. Retrieval evaluation asks whether the evidence set was found and correctly ordered. Answer evaluation asks whether a response stayed within that evidence. Combining them into one subjective score hides the failure owner: a perfect answer cannot repair missing evidence, and good retrieval does not guarantee a grounded answer. The original retrieval-augmented generation paper describes generation conditioned on retrieved non-parametric memory; that separation is a useful conceptual boundary even when the production components differ.

Include ordinary positive cases, but spend more design effort on contrasts. “Does the lunch menu have ramen?” should be paired with a dinner-only item, an unavailable item with the same name, and the same item at another restaurant. Add queries that should return no evidence. Otherwise a retriever that always returns something can look healthy while producing the most dangerous kind of confident mistake.

One long case is worth spelling out. Suppose a restaurant publishes an item as available for lunch, later marks it unavailable, and keeps the description unchanged. The evaluation set needs separate source versions with non-overlapping effective intervals. A query evaluated after the change must forbid the earlier item version even though its lexical match may be perfect. If the stale version ranks first, record stale_at_1; if the current unavailable record is found, the assistant has evidence to decline rather than improvise. This catches a temporal architecture failure that a generic relevance label misses.

Build a versioned evaluation set, not a static spreadsheet

Store cases in reviewable data with stable identifiers. The format should be boring enough for menu operators to inspect and precise enough for an evaluator to execute. A minimal record might look like this:

EVALUATION_CASES = [
    {
        "case_id": "lunch-availability-001",
        "query": "Is the mushroom ramen available for lunch?",
        "as_of": "2026-08-30T12:00:00+08:00",
        "restaurant_id": "restaurant-17",
        "expected_ids": {"item-884-v3"},
        "forbidden_ids": {"item-884-v2", "item-884-other-location"},
        "facets": {
            "question_type": "availability",
            "service_window": "lunch",
            "locale": "en-US",
        },
    }
]

The timestamp is an input to retrieval, not decorative metadata. At query time, filter candidates to records whose restaurant and effective interval apply before ranking text similarity. A ranking stage should not have to overcome records that were never eligible. This also gives the evaluation set a clean failure taxonomy: eligibility filtering, candidate recall, ranking, or answer grounding.

Freshness is eligibility.

Version the case file whenever expected evidence changes, and link that version to the source snapshot. Don't silently rewrite the expected ID after a menu edit; doing so erases the distinction between a legitimate content change and a retrieval regression. When labels are genuinely ambiguous, allow a set of acceptable IDs and record the unresolved editorial question. Your mileage may vary on who approves those labels, but the owner should be named in the repository process.

Measure freshness and ranking separately

Recall at a small cutoff answers one narrow question: did at least one acceptable record appear? It does not penalize a stale record above the correct one, so add a forbidden-hit measure and inspect rank one directly. The evaluator below expects retrieval results already filtered by the case timestamp; it calculates outcomes without calling a model.

from dataclasses import dataclass
from typing import Iterable


@dataclass(frozen=True)
class Result:
    item_id: str
    score: float


def evaluate_case(
    expected_ids: set[str],
    forbidden_ids: set[str],
    results: Iterable[Result],
    cutoff: int = 5,
) -> dict[str, object]:
    ranked = list(results)[:cutoff]
    ranked_ids = [result.item_id for result in ranked]
    expected_ranks = [
        rank
        for rank, item_id in enumerate(ranked_ids, start=1)
        if item_id in expected_ids
    ]
    forbidden_ranks = [
        rank
        for rank, item_id in enumerate(ranked_ids, start=1)
        if item_id in forbidden_ids
    ]

    return {
        "recall_at_k": int(bool(expected_ranks)),
        "first_expected_rank": min(expected_ranks, default=None),
        "forbidden_hit_at_k": int(bool(forbidden_ranks)),
        "stale_at_1": int(bool(ranked_ids) and ranked_ids[0] in forbidden_ids),
    }

A case with no expected evidence needs a different assertion: the eligible result set should be empty, or every score should fall below a threshold calibrated on labeled cases. Do not fold these cases into ordinary recall because a no-answer contract has no relevant document to recall. Track them as rejection cases, and review threshold changes against positives and rejections together.

Aggregate metrics can conceal a restaurant or facet that is completely broken. Report macro results by restaurant and question type, then keep the underlying case failures. A single overall pass rate is especially misleading when availability questions are rare but consequential. No drama. Slice the data.

Turn evaluation failures into deployment gates

Run a focused suite when a menu snapshot changes: select cases referencing changed item IDs, plus cases for the same section, restaurant, and temporal boundary. Run the entire suite on a scheduled cadence and before changes to chunk construction, eligibility filters, ranking, or embedding configuration. Compare against a checked-in baseline, but gate on explicit safety invariants as well as aggregate movement.

Consider the deployment path for one item whose dinner availability begins at 17:00. The ingest job writes a new immutable version and closes the prior version's effective interval; it does not overwrite the earlier record. The focused case selector finds cases tied to that item, its menu section, and the 17:00 boundary. Retrieval at 16:59 must exclude the dinner version, retrieval at 17:00 may include it, and a query against another restaurant must exclude both versions. The evaluator records returned version IDs before any answer is generated, which means an eligibility failure cannot be disguised by fluent wording. If those checks pass, the answer layer can be tested against the retrieved text as a separate gate. If they fail, deployment stops with a case ID, snapshot hash, and retrieval configuration that an engineer can replay. This is more retained detail than a pass-rate dashboard, but far less than keeping every successful conversation indefinitely, and each retained field has a diagnostic job.

Boundaries first.

For example, a release can require zero forbidden records at rank one for allergen and availability cases, while allowing a reviewed tolerance for recall changes in descriptive discovery queries. The exact tolerance is a policy decision, not a transferable fact. Record the case IDs that moved, inspect the retrieved source versions, and require a label review when the expected evidence changed for legitimate menu reasons.

Observability should use the same identifiers as evaluation: menu snapshot, chunk version, retrieval configuration, case or query trace, and returned item versions. This makes an online miss reproducible as a new offline case without retaining every prompt forever. Redact or avoid diner-provided personal data before storage, set retention by data class, and test deletion as an operational path rather than a policy sentence.

This architecture is not suitable when the menu source cannot provide stable item identity or effective timestamps. In that case, improve the publishing contract first; adding semantic ranking will only make stale records harder to diagnose. A static keyword index may also be the better choice for a small, exact-name menu with few paraphrases, because its behavior is easier to inspect. Use richer retrieval only after the evaluation set demonstrates a miss that richer retrieval can plausibly fix.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401

Further reading

• https://arxiv.org/abs/2005.11401