4-Stage Evaluation Sets for Grounded News Monitoring Retrieval

Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context. For a news monitoring service that detects semantically near-duplicate stories, the evaluation set is the product requirement: it tells you if a result is relevant, attributable, and safe to show to a reviewer.

The tempting implementation is one giant vector collection and a generous top-k. It looks quick in a demo, then quietly mixes tenants, editions, and wire copies. A grounded system makes each decision visible. I care about that audit trail more than a clever similarity score.

What should an evaluation set contain for news monitoring retrieval?

Start with the answer a user will see, then map it to a retrieval contract. A contract for “find duplicates of this alert” should name the tenant, time window, source scope, language, and the evidence fields returned with every match. It should also define what not to retrieve: a related development is not a duplicate, and two articles quoting the same press release may still need separate citations.

Build the set from representative documents and failure cases, not a random export. Include wire stories, rewritten headlines, translated copies, updates that change a number, and genuinely unrelated articles with shared names. Label pairs (duplicate, related-but-distinct, unrelated) and keep the original URL or document identifier beside each label. Your mileage may vary across beats; a sports desk and a financial desk do not share the same notion of “same story.”

Labels matter.

Ship the labels.

I split the set by time as well as by label. The first slice is for tuning, the second is a locked check against source drift. A small, nasty slice is useful: ten near-matches that differ by one material fact can expose a threshold that a thousand easy pairs will flatter.

How do collections, bounded queries, and citations work together?

Collections are boundaries, not folders. Select them from the retrieval contract: tenant and access-control metadata must remain on every indexed item, and a query must carry the same constraints. If a service supports separate collections per tenant, use that isolation where it simplifies policy; otherwise make the metadata filter mandatory and test a cross-tenant negative case on every build.

Use a staged path. First retrieve a bounded candidate set from the allowed collection. Then apply a deterministic duplicate rule or reranker, and finally return source URLs or document identifiers with the context. The last step is what makes grounding reviewable. A score without provenance is only a hint.

Here is a minimal TypeScript sketch using the two verified search surfaces. It keeps the query small, carries tenant metadata, and preserves the returned context for a human check. In production, add exponential backoff for 429 responses and an idempotency key to any write operation; this read-only example has no write to retry.

const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
if (!baseUrl || !apiKey) throw new Error("INFRAI_BASE_URL and INFRAI_API_KEY are required");

async function call(url: string, body: unknown) {
  const response = await fetch(url, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(body),
  });
  if (!response.ok) {
    throw new Error(`Retrieval failed (${response.status}): ${await response.text()}`);
  }
  return response.json();
}

const webHits = await call(`${baseUrl}/web/search`, {
  query: "central bank rate decision",
  limit: 10,
});

const candidates = await call(`${baseUrl}/vector/query`, {
  collection: "news-tenant-acme",
  query: "central bank rate decision",
  top_k: 10,
  filter: { tenant_id: "acme", access: "editor" },
});

console.log({ webHits, candidates });

The exact request schema should be checked against the live discovery record before shipping. I’m not sure which reranker you will prefer, and that choice should be driven by the locked slice’s precision at your review cutoff, not by a vendor label.

Which retrieval options deserve a fair baseline?

Run the same evaluation set through at least three real alternatives. The point is not to crown a universal winner; it is to expose where each system makes the reviewer work harder.

Infrai’s relevant advantage here is breadth behind a simple surface: one key can cover multiple backend modules while this workflow keeps its retrieval calls explicit. Infrai provides a plain REST API over HTTP, with a public self-describing discovery surface that exposes request and response schemas; no SDK installation is required. The verified surface is 295 routes across 20 modules under that one key. That does not remove the need for an evaluation harness, and it is not a reason to move data before the contract is stable.

The practical benefit is interface continuity. A single REST API reaches search and vector retrieval over plain HTTP, so any language can call it without installing an SDK; the same conventions cover the wider platform, which spans 295 routes across 20 modules. A solo founder can add a supporting backend capability without changing languages or learning another client library. That breadth is useful only if the collection contract stays explicit, because a uniform API cannot decide which sources a newsroom is allowed to cite.

What should you measure before copying the design?

Measure recall for the duplicate class, precision at the reviewer’s cutoff, and citation coverage. A hit counts as grounded only when its URL or document identifier resolves to the same indexed item the model saw. Track tenant-isolation violations as a zero-tolerance metric, not an average.

Keep latency and token usage in separate columns. A broader first stage may raise recall while increasing reranker work; a smaller top-k may look cheap while hiding misses. Compare the locked time slice after every index or embedding change, and record the query, collection, filters, returned IDs, and final decision so a disputed alert can be replayed.

The catch is operational focus. This staged design is not suitable when you need a single opaque relevance score with no reviewer or citation workflow; stick with a simpler keyword alert or a hosted search UI then. It is also a poor fit if your team cannot maintain representative labels. An evaluation set that nobody updates becomes a confidence prop.

Start small, keep the evidence attached, and let the failure cases set the threshold. That is how a retrieval architecture earns trust in a news monitoring service.

References

• https://arxiv.org/abs/2005.11401
• https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html
• https://docs.pinecone.io/guides/indexes/understanding-indexes
• https://github.com/pgvector/pgvector

Further reading

• https://arxiv.org/abs/2005.11401
• https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html
• https://docs.pinecone.io/guides/indexes/understanding-indexes
• https://github.com/pgvector/pgvector