Insurance Claims Retrieval Architecture: Access Rules for Fresh, Auditable Intake

Insurance claims systems have a hard constraint: a retrieved fact is only useful if the claimant, adjuster, and model were allowed to see it at that moment. Short answer: enforce access rules before retrieval, keep chunks small enough to revoke, and make freshness plus authorization part of the retrieval SLO.

I learned this while operating an e-commerce change watcher, a smaller system with the same dangerous shape. We fetched product pages, chunked the text, and alerted on diffs. A cache refresh left a discontinued promotion in the candidate set for 17 minutes; the diff was accurate against the stale copy, but the alert was operationally wrong. In claims intake, the equivalent mistake is returning an old coverage endorsement or a note from a different policyholder. The model can write a fluent answer and still be violating the access contract.

That incident changed my default design. Retrieval is a policy decision with a search operation inside it, not a search feature with a permission check bolted on later. The alert pipeline had looked healthy: fetch latency was under its target, the queue had drained, and the diff worker had no retries. The missing signal was semantic freshness. We had invalidated the page cache, but not the derived chunks that fed the diff comparison. During the postmortem, the fix was less interesting than the invariant: every derived record needs an owner, a version, and a measurable deadline for becoming ineligible. Claims intake has more severe consequences, because an old deductible can change a payment decision rather than just trigger a noisy notification.

No exceptions.

How should retrieval architecture enforce access rules for insurance claims intake?

Start with an authorization context that is explicit and immutable for one request: tenant, claim ID, actor role, jurisdiction, and an authorization-version or policy timestamp. Resolve that context against the system of record, then pass only permitted document IDs into the retriever. Filtering after vector similarity is too late; an unauthorized chunk can influence ranking, logs, or a generated answer even if it is removed before display.

Every chunk needs provenance fields that survive embedding and storage: claim_id, policy_id, tenant_id, document_version, effective_from, effective_to, classification, and a hash of the source. A parent document can be large, but the authorization unit should be small. If a medical invoice is rescinded, revoking that invoice's chunk should not require rebuilding an entire claimant's index.

The request path should fail closed. Missing policy context means no retrieval, not a broad default scope. A denied result should be indistinguishable from an empty result to the language model, while the audit stream records the reason, actor, and policy version for human review.

The incident pattern: freshness turns a correct answer into a wrong decision

Chunking and freshness are coupled. Large chunks reduce index overhead but keep revoked text alive longer and make precise citations awkward. Tiny chunks improve revocation and citation boundaries, yet they can split a coverage condition from its exception. I use a parent-child scheme: embed 200–500 token children, retain a parent section for display, and attach the parent's effective dates to every child. Those ranges are an engineering starting point, not a universal benchmark; your mileage may vary with document layout and OCR quality.

For claims, freshness should be measured as policy lag: the time between a source update or revocation and the last retrieval that could return the old version. Set a target per document class. A newly uploaded loss notice may need a five-minute target; an archived weather bulletin might tolerate a day. The watcher incident had no such class-specific SLO, so its green ingestion metric concealed a red decision metric.

Use a tombstone index or version filter so old chunks are excluded immediately, then clean them asynchronously. Re-embedding is not authorization. A fresh vector for a revoked note is still forbidden.

That distinction is easy to lose under delivery pressure. A backfill job may report success while a replica still serves an earlier version, and a dashboard may show index age without showing policy age. Put both values on the same trace and alert on the larger one. The operator needs to know whether the next action is to repair ingestion, refresh policy state, or stop automated intake.

A preventative request path in Go

The following sketch keeps the security boundary visible. The interfaces are deliberately generic so the same path can sit over a database, a self-hosted vector index, or a managed search service.

package intake

import (
    "context"
    "errors"
)

type Authz struct {
    TenantID   string
    ClaimID    string
    ActorID    string
    Role       string
    PolicyRev  string
}

type Chunk struct {
    ID, Text, ClaimID, TenantID, Version string
}

type Policy interface {
    Resolve(ctx context.Context, actorID, claimID string) (Authz, error)
}

type Retriever interface {
    Search(ctx context.Context, query string, scope Authz, limit int) ([]Chunk, error)
}

type Answerer interface {
    Compose(ctx context.Context, question string, evidence []Chunk) (string, error)
}

func Answer(ctx context.Context, p Policy, r Retriever, a Answerer,
    actorID, claimID, question string) (string, error) {
    scope, err := p.Resolve(ctx, actorID, claimID)
    if err != nil {
        return "", err
    }
    if scope.TenantID == "" || scope.ClaimID != claimID || scope.PolicyRev == "" {
        return "", errors.New("authorization context is incomplete")
    }

    // The retriever must apply tenant, claim, role, classification, and version
    // filters before scoring candidates.
    evidence, err := r.Search(ctx, question, scope, 8)
    if err != nil {
        return "", err
    }
    return a.Compose(ctx, question, evidence)
}

The important contract is in Retriever.Search: its implementation must reject a scope mismatch and apply the policy revision before similarity scoring. Log the query fingerprint, returned chunk IDs, document versions, freshness age, and authorization revision. Do not log raw medical text by default. For an SRE, those fields make a replayable audit without turning observability storage into a second claims database.

Choosing an operating model without hiding the trade-offs

The platform roadmap usually comes down to who owns indexing, policy enforcement, and the pager. A managed search service can reduce operational toil, but it does not transfer responsibility for access semantics; you still have to prove that filters are applied before ranking and that deletes propagate within the freshness SLO.

The catch is that this architecture is not suitable when claims data must be searched across tenants with no stable policy identity, or when the source cannot provide reliable effective dates. In those cases, stick with a transactional, human-reviewed workflow until those controls exist. A vector index cannot manufacture a trustworthy access rule.

Capacity planning belongs in the same review. Estimate peak claim submissions, average chunks per claim, embedding throughput, and the maximum backfill window after a policy change. Keep headroom for a burst, then load-test the authorization filter itself; a fast similarity query that waits on a policy database still misses the intake SLO.

Verification, recovery, and the limits of generation

Test access as a matrix, not a happy path: claimant versus adjuster, active versus closed claim, current versus superseded endorsement, and cross-tenant IDs that share vocabulary. Seed each case with a canary chunk and assert it is absent from both retrieval results and model citations when denied. Run the suite on every schema or policy change.

Measure retrieval recall only inside the authorized set. Pair it with citation coverage, policy lag, denied-query rate, and p95 end-to-end latency. If a source update is delayed, surface “evidence is stale” to the intake operator and route the claim for review; do not ask the model to guess. I'm not sure any single freshness number will fit every line of business, so document the class-specific rationale and revisit it with incident data.

The research foundation for retrieval-augmented generation is useful here because it separates retrieved evidence from generation, but it does not define your tenant policy or retention obligations. Those remain application responsibilities. Keep the boundary explicit, keep the evidence versioned, and let the model operate only on chunks the request was already entitled to read.

References

• https://arxiv.org/abs/2005.11401
• https://www.nist.gov/privacy-framework
• https://www.rfc-editor.org/rfc/rfc9110