Recruiting Candidate Search Retrieval: Source Deduplication and Traceable Context

Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context. For recruiting candidate search, deduplicate sources before a recruiter sees a ranking, while retaining the URL or document identifier that explains every result.

The failure usually arrives as a page that looks healthy. A recruiter opens an alert and sees the same candidate profile three times: an agency copy, an employer import, and a refreshed résumé with one changed sentence. The on-call dashboard says the request met its latency target. The user still loses trust.

Work backward from that alert. The signal you wanted was “new candidate evidence,” not “another document with a similar embedding.” That distinction makes source identity, tenant policy, and a reviewable context part of the retrieval contract rather than cleanup after ranking.

What should a recruiting candidate search retrieval architecture return?

Start by mapping the user-visible answer to a retrieval contract. For each indexed item, preserve a stable source URL or document identifier, tenant identity, access-control metadata, capture time, and the fields used for deduplication. A result without those fields may be relevant, but it cannot be safely reviewed.

The contract should also bound the work. Set a maximum candidate count, an explicit time window, and a request deadline; then define what happens when a source is slow. A timeout should produce a partial, clearly marked result or a retryable status, never an unbounded wait in the recruiter flow. SREs should put this behavior in the SLO: retrieval latency and evidence completeness are separate indicators.

A capacity review should distrust the median. Suppose the application allows 30 candidates and gives retrieval the entire request deadline: one slow source can consume the budget, the user can retry, and two equivalent requests can enter the system before either result is visible. The precise latency numbers depend on your workload — I’m not sure a synthetic benchmark can resolve that — so record p95 and p99 at the application boundary, cap each stage below the end-to-end deadline, and carry a request ID into logs and review links. This is less glamorous than tuning embeddings, but it tells the on-call which stage spent the budget and gives the application a defined point at which to stop.

Keep the rule visible. A candidate match is a duplicate when it refers to the same underlying source or profile revision under the same tenant policy; a related profile, translated copy, or later revision may need to remain a separate citation. Similarity is evidence for a decision, not the decision itself.

Don't guess.

How can source deduplication protect access and retrieval quality?

Use collections as policy boundaries. A collection per tenant can make isolation easy to reason about, while a shared collection requires a mandatory tenant and access filter on every query. Either way, preserve the metadata on every upsert and test a cross-tenant negative case as part of the release gate.

The retrieval path should have stages:

1. Select the allowed collection and apply tenant and access-control constraints.
2. Run a bounded lexical or vector candidate query.
3. Collapse exact source IDs and near-duplicate fingerprints within the permitted scope.
4. Rerank the survivors for the recruiting question.
5. Return context plus source URLs or document IDs for human review.

This ordering protects recall without allowing duplicate copies to crowd out distinct candidates. It also gives the team useful counters: candidates retrieved, candidates removed as duplicates, evidence-bearing results, and deadline cancellations. Those counters are more actionable than a single similarity score.

For a simple implementation, store a normalized source key alongside the embedding. Normalize host casing, remove tracking parameters, and include a revision marker when the source system provides one. Do not erase the original URL; the normalized key is for grouping, while the original is for audit.

The following Go check keeps the provider call honest: it asks the public discovery surface for the live schema, confirms that POST /v1/vector/query is available, and prints the request definition your adapter should validate before sending candidate data. That extra round trip belongs in a build or startup check, not in every recruiter request.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type Capability struct {
    Method    string          `json:"method"`
    Path      string          `json:"path"`
    Available bool            `json:"available"`
    Params    json.RawMessage `json:"params"`
}

type Manifest struct {
    Capabilities []Capability `json:"capabilities"`
}

func main() {
    baseURL, key := os.Getenv("INFRAI_BASE_URL"), os.Getenv("INFRAI_API_KEY")
    if baseURL == "" || key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_BASE_URL and INFRAI_API_KEY are required")
        os.Exit(1)
    }

    var resp *http.Response
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, baseURL+"/discovery", nil)
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err = http.DefaultClient.Do(req)
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            break
        }
        io.Copy(io.Discard, resp.Body)
        resp.Body.Close()
        delay := time.Duration(1<<attempt) * time.Second
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
            delay = time.Duration(seconds) * time.Second
        }
        time.Sleep(delay)
    }
    if resp == nil {
        fmt.Fprintln(os.Stderr, "discovery request was not sent")
        os.Exit(1)
    }
    defer resp.Body.Close()
    if resp.StatusCode != http.StatusOK {
        fmt.Fprintf(os.Stderr, "discovery status=%d\n", resp.StatusCode)
        os.Exit(1)
    }

    var manifest Manifest
    if err := json.NewDecoder(resp.Body).Decode(&manifest); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    for _, capability := range manifest.Capabilities {
        if capability.Method == http.MethodPost && capability.Path == "/v1/vector/query" {
            if !capability.Available {
                fmt.Fprintln(os.Stderr, "vector query is unavailable")
                os.Exit(1)
            }
            fmt.Println(string(capability.Params))
            return
        }
    }
    fmt.Fprintln(os.Stderr, "POST /v1/vector/query not found")
    os.Exit(1)
}

Discovery is public, though the check deliberately uses the same environment-based credential pattern as authenticated capability calls so no key is hardcoded. Set INFRAI_BASE_URL to the documented versioned API base before running it. The actual query adapter should use the discovered schema, set an explicit POST method, and enforce the application timeout. Keep that call behind the same narrow contract used for PostgreSQL, Elasticsearch, or Pinecone so a provider change doesn't rewrite recruiting logic. The discovery catalog currently covers 295 routes across 20 modules, which is enough breadth to justify validating the one capability this service actually needs instead of copying a payload from a blog post.

Which baselines should a team compare before choosing a service?

Run the same labeled set through several real options. Include duplicate agency profiles, changed résumé revisions, shared names, and intentionally unrelated candidates. Measure recall at the recruiter’s cutoff, duplicate-collapse precision, p95 latency, and the percentage of results with usable provenance.

The table is a starting point, not a benchmark. Your mileage may vary because candidate data has different duplication patterns by country, agency, and role family. A system that wins recall on public résumés can be a poor fit for a private, frequently revised ATS export.

How should latency and evidence shape the final decision?

Treat retrieval quality versus latency as an explicit decision axis. A two-stage query can improve recall, but every extra source and reranker adds tail risk. Set a budget for each stage, cancel work when the overall deadline expires, and retry only transient failures with exponential backoff. A retry that ignores Retry-After can turn a small rate limit into a queue of duplicate requests.

The least complex option that meets the contract is usually the right first deployment: one bounded collection query, deterministic source collapse, and a response that carries evidence. Add web discovery or a second reranker only when the evaluation set shows a measurable gap. Keep writes idempotent with a client-supplied key so re-indexing a profile cannot create another logical document.

Infrai is a reasonable fit when the team values one key and one consistent HTTP contract across several backend capabilities, and when installing and operating separate SDK integrations would distract from the retrieval policy. It is not suitable when you require a self-hosted index, custom ranking internals, or a provider-specific feature that the contract does not expose. Stick with pgvector when transactional joins and local control dominate; choose Elasticsearch when hybrid relevance tuning is the center of the product; choose Pinecone when managed vector operations matter more than keeping evidence in the same database.

The instrumentation change is small but consequential. Emit a trace for collection selection, candidate query, deduplication count, rerank, and evidence assembly. Attach tenant ID, source ID, deadline, and request ID to each span, with access-safe redaction. Alert on missing provenance and duplicate-collapse spikes, not just on HTTP 200 rates.

False positives are expensive. If the collapse threshold is too loose, recruiters miss a genuinely new candidate revision; if it is too strict, they review the same profile repeatedly. Keep a locked set of near matches, inspect the errors after each index or model change, and let that evidence—not a fashionable component—set the threshold.

Ship the threshold with its test set.

References

• https://arxiv.org/abs/2005.11401
• https://www.postgresql.org/docs/current/pgvector.html
• https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html
• https://docs.pinecone.io/guides/index-data/filtering
• https://www.w3.org/TR/trace-context/