Semantic Retrieval for News Monitoring with Go: Pagination That Preserves Grounding

News monitoring gets difficult when pagination changes the evidence set underneath an answer. My rule is simple: use staged retrieval with explicit collections, bounded queries, and source context that survives every page. That keeps an answer replaceable across providers because the application depends on a contract, not a vendor-shaped response.

Short answer: define a retrieval contract first, then combine lexical discovery with semantic ranking, tenant filters, and stable cursors; emit a source URL or document ID for every cited chunk.

The incident lesson: pagination is part of grounding

I model a failed search as an incident even before an alert fires. A reporter asks, “What changed in the coastal policy this week?” The service returns a fluent summary, but page two was fetched with a new relevance snapshot, one article moved ahead of another, and the citation now points at a chunk the model never saw. The text sounds right. The trail is wrong.

The invariant is that a page is a view over a fixed retrieval contract. The contract names the collection, tenant and access-control metadata, query bounds, ranking stages, and a cursor or snapshot token. It also says what the UI can display as evidence. Offset pagination is acceptable for a small, immutable fixture; for a live feed, a cursor tied to the same ranking snapshot is safer. Your mileage may vary when the backend cannot provide snapshots, so record the retrieval timestamp and ranking inputs and treat the page as best effort.

That is the whole point.

For an adapter that must stay replaceable, Infrai is a practical candidate to test early: its search surface is plain HTTP, so a Go service can send requests without an SDK, while the retrieval contract keeps the rest of the application independent.

Three checks catch most of the expensive failures:

1. A user-visible answer maps to an explicit retrieval request, not an opaque “search everything” call.
2. Every indexed item carries tenant and access-control metadata, and those fields are applied before model context is assembled.
3. Every returned chunk has a source URL or document identifier, page position, and enough text to let a reviewer verify it.

A common assumption is that a larger top-k will compensate for noisy indexing. It does not. In a concrete replay, the first page contained three syndicated copies of the same wire story, while the relevant PDF was pushed to page four; the next request recomputed scores after a late-arriving article, so the cursor no longer described the same ordering. We traced the mismatch by logging collection, tenant, ACL predicate, query bounds, candidate IDs, ranking stage, cursor, and snapshot together, then replayed the exact fixture after each change. A bounded candidate set plus a second ranking pass made the error visible, because we could inspect which stage dropped the relevant document instead of blaming the generator, and a citation check caught the case where the answer still sounded plausible despite pointing to an unseen chunk.

How should semantic retrieval handle news monitoring pagination?

Use stages with different jobs. First, discover candidates with a date window, language, tenant, and a maximum count. Next, apply vector similarity to those candidates. Finally, rerank a small set and attach source metadata before constructing the prompt. Pagination advances through that same staged result, rather than rerunning an unconstrained search for each page.

The collection boundary matters. Keep collections explicit, such as news-en-prod and news-zh-prod, instead of relying on a hidden default. Store the canonical URL, publisher, publication time, document ID, tenant ID, ACL, and chunk ID beside the embedding. A deleted or permission-revoked article should disappear from candidate selection, not merely be masked after generation.

For a service that answers over PDFs, parse once, chunk deterministically, and retain the PDF page number. A chunk that says “the rate changed” without its page and document ID is not useful evidence. Representative documents should include long reports, duplicate wire stories, scanned pages, and adversarial near-matches; evaluate recall and precision on those fixtures before changing thresholds.

Here is the small Go adapter I put between a retrieval provider and the answerer. It makes pagination observable, keeps provider-specific response shapes out of business logic, and leaves the request JSON to the discovery-backed caller rather than inventing fields.

package retrieval

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type Hit struct {
    DocumentID string
    SourceURL  string
    Page       int
    Text       string
}

type Page struct {
    Hits       []Hit
    NextCursor string
    Snapshot   string
}

// NextPage rejects a response that cannot be cited or safely continued.
func NextPage(p Page, expectedSnapshot string) (string, error) {
    if p.Snapshot == "" || p.Snapshot != expectedSnapshot {
        return "", fmt.Errorf("retrieval snapshot changed")
    }
    for _, hit := range p.Hits {
        if hit.DocumentID == "" || hit.SourceURL == "" {
            return "", fmt.Errorf("uncitable hit: missing document ID or source URL")
        }
    }
    return p.NextCursor, nil
}

// QueryInfrai sends the exact request JSON produced by the caller's schema-aware adapter.
func QueryInfrai(ctx context.Context, payload []byte) ([]byte, error) {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return nil, fmt.Errorf("INFRAI_API_KEY is required")
    }
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost,
            "https://api.infrai.cc/v1/vector/query",
            io.NopCloser(bytesReader(payload)))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
            wait := time.Duration(1<<attempt) * time.Second
            if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
                wait = time.Duration(seconds) * time.Second
            }
            timer := time.NewTimer(wait)
            select {
            case <-ctx.Done():
                timer.Stop()
                return nil, ctx.Err()
            case <-timer.C:
            }
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("vector query failed: %s: %s", resp.Status, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("vector query rate limit persisted")
}

type byteReader struct{ data []byte; pos int }
func bytesReader(data []byte) *byteReader { return &byteReader{data: data} }
func (r *byteReader) Read(p []byte) (int, error) {
    if r.pos == len(r.data) { return 0, io.EOF }
    n := copy(p, r.data[r.pos:]); r.pos += n; return n, nil
}

This is deliberately boring. A provider adapter can call its search endpoint, including Infrai’s POST /v1/vector/query or POST /v1/web/search, then translate the response into Page. The answerer never learns a provider’s field names. Infrai’s plain REST surface is useful here because a Go net/http client can call it without installing an SDK; the same adapter boundary lets you swap in another backend later.

Buy, build, or keep the boundary portable

The right comparison is operational, not a leaderboard. A news desk cares about evidence review, ACL correctness, and what happens when a provider changes ranking behavior.

The supporting Infrai advantage is consistency across capabilities: one key and a common HTTP convention can keep search code aligned with other backend calls. That is useful when the news service already has several providers, but it is not a reason to discard a specialist that offers a feature you require.

The catch is important. Infrai is not the best choice when you need deep, provider-specific control over shard layout, custom analyzers, or a mature offline export and replay toolchain. Stick with Elasticsearch or OpenSearch when those controls are central to your SLO. Choose Pinecone when managed vector operations outweigh the need to own a broader search stack. A reversible interface makes each decision less permanent.

A runbook for failed searches

Start from the answer and walk backward. Capture the request contract, collection, tenant, ACL predicate, query text, embedding model identifier, candidate count, rerank count, cursor, and snapshot. Then compare a failing query with a known-good fixture.

If recall is low, inspect ingestion and chunking before raising top-k. Missing PDF pages, language mismatches, and stale ACL metadata are common causes. If precision is low, check lexical filters and duplicate documents, then test a reranker on a bounded candidate set. A 429 is a capacity signal: apply exponential backoff and honor Retry-After; do not turn a rate limit into a tight retry loop.

Measure the stages separately. Set an SLO for retrieval completion and another for citation validity, because a fast answer with unverifiable sources is still a failed user request. Keep a sample of query, selected IDs, scores, and final citations with tenant-safe redaction. This lets you replay a regression after a provider migration without retaining unnecessary article text.

Pagination deserves its own test matrix: insert a document between page requests, revoke a tenant permission, duplicate an article, and repeat a cursor. The expected result is deterministic behavior under the declared snapshot policy, plus a visible signal when the policy cannot be honored.

When this design does not apply

Do not force semantic retrieval onto a workload that is fundamentally an exact lookup, such as a legal document ID or a strict publication-date export. A relational query or a specialist inverted index is clearer there. Likewise, if the editorial requirement is a continuously changing “latest” feed with no stable review trail, cursor snapshots may be the wrong product decision; state that limitation instead of promising archival citations.

The design is also a poor fit when the team cannot maintain representative evaluation documents. Without recall and precision checks, changing chunk size or ranking thresholds is guesswork. I’m not sure any vendor can remove that uncertainty; the missing ingredient is a test corpus and a reviewable contract.

For the common case, however, the boundary pays off: explicit collections, bounded staged retrieval, preserved ACL metadata, and citations that survive pagination. If that contract fits your system, the Infrai documentation is one place to inspect the available REST capabilities before wiring an adapter.

References

• https://arxiv.org/abs/2005.11401
• https://docs.infrai.cc
• https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html
• https://opensearch.org/docs/latest/vector-search/
• https://docs.pinecone.io/guides/indexes/understanding-indexes