How to Keep Recruiting Search Fresh — Bounded Retrieval Under Source Failure

Short answer: use staged retrieval with explicit collections, hard latency budgets, and source context on every result; when fresh web evidence misses its budget, return a clearly labeled indexed result instead of blocking candidate search.

Consider a B2B recruiting SaaS that watches a candidate's public portfolio page and alerts a recruiter when the page changes. The product has two promises that pull against each other. A recruiter wants a fast result now, but a fresh page diff may contain the strongest evidence. Waiting indefinitely protects neither promise.

My default is staged retrieval. Keep the candidate index on the critical path, put page refresh behind a smaller deadline, and preserve enough provenance to explain which version produced the result. That choice is about revenue per engineering hour: the team can ship the alert while outsourcing undifferentiated retrieval plumbing, yet the user never waits forever on somebody else's page.

Set a freshness budget before choosing a backend

Start with the product decision. For each search, decide how old evidence may be before the UI must say so, how long a fresh lookup may consume, and whether an indexed answer is still useful after that deadline. Those are separate values. A 900 ms web budget doesn't imply that nine-hour-old evidence is acceptable, and a fresh page doesn't justify a five-second blank screen.

The retrieval contract should make four things explicit: collection identity, tenant and access-control metadata, a source URL or document identifier, and the time at which the source was observed. A collection named for a tenant is helpful, but it isn't a substitute for carrying tenant and permission scope on every indexed item. The filter belongs before ranking. Cross-tenant evidence is not a low-quality result; it is a broken authorization boundary.

For the page-watching job, I would store the normalized page text, its digest, its source URL, the candidate ID, the tenant ID, the access scope, and the observation time in the application's durable ledger. On a changed digest, the worker can upsert the new evidence into the selected collection and generate an alert whose link points back to the reviewed source. On an unchanged digest, it should do nothing. This keeps “no change” cheap in product complexity even when the retrieval provider charges or behaves differently.

One rule matters most: a stage may improve an answer, but it may not silently redefine who is allowed to see it.

Scope first.

How should recruiting candidate search recover from retrieval source failure?

Model recovery as states, not as a pile of catch blocks. A request begins with indexed; it can become fresh when the watched page returns within budget, stale when the stored evidence is still allowed but the refresh misses its deadline, or unavailable when neither source satisfies the contract. The UI should receive that state with the evidence. Hiding it turns a latency decision into an unexplained ranking change.

Retry narrowly. HTTP 429 is the concrete case: honor Retry-After, add exponential backoff when the header is absent, and stop at a fixed attempt count inside the stage deadline. Authentication and validation failures should surface immediately rather than consume the retry budget. I'm not sure one timeout will fit every recruiting workflow; a search box and an audit export have different patience. The way to resolve that uncertainty is to measure time to first useful result and citation-open rate for each workflow, then set separate budgets.

Keep the fallback boring. If the live page stage expires, query the explicit candidate collection with a bounded result limit, attach the last observation time, and label the evidence stale when policy requires it. If the indexed query also fails its contract, return unavailable. Don't fabricate a summary from missing context, and don't broaden the query to another tenant or collection just to fill the screen.

This is where traceability earns its keep. Record the stage, attempt count, elapsed time, collection, source document IDs, and final recovery state. A recruiter asking “why did this person match?” needs the evidence that actually participated in that response, not whatever the page says during a later replay.

Run the recovery policy as a small TypeScript program

The following program is intentionally provider-neutral. It runs end to end, injects one 429, applies bounded backoff, detects a portfolio-page diff, and returns source context with the candidate result. Replace the two mock adapters with your chosen services; keep the contract and deadline behavior intact.

type AccessScope = "recruiter" | "hiring-manager";

type Evidence = {
  candidateId: string;
  tenantId: string;
  access: AccessScope[];
  sourceUrl: string;
  documentId: string;
  observedAt: string;
  digest: string;
  text: string;
};

type RetrievalResult = {
  state: "fresh" | "stale" | "unavailable";
  evidence: Evidence[];
  attempts: number;
  reason?: string;
};

type VectorQuery = {
  collection: string;
  query: string;
  limit: number;
};

class RateLimitError extends Error {
  constructor(readonly retryAfterMs: number) {
    super("rate limited");
  }
}

const indexed: Evidence = {
  candidateId: "cand_2048",
  tenantId: "tenant_northwind",
  access: ["recruiter"],
  sourceUrl: "https://portfolio.example/candidates/2048",
  documentId: "portfolio_2048_v7",
  observedAt: "2026-09-03T08:00:00Z",
  digest: "sha256:old",
  text: "Led an invoicing migration for a B2B product."
};

let liveAttempts = 0;

async function fetchWatchedPage(): Promise<Evidence> {
  liveAttempts += 1;
  if (liveAttempts === 1) throw new RateLimitError(20);
  return {
    ...indexed,
    documentId: "portfolio_2048_v8",
    observedAt: "2026-09-03T08:15:00Z",
    digest: "sha256:new",
    text: "Led an invoicing migration and documented rollback drills."
  };
}

async function queryIndex(
  tenantId: string,
  access: AccessScope,
  limit: number
): Promise<Evidence[]> {
  const allowed =
    indexed.tenantId === tenantId && indexed.access.includes(access);
  return allowed ? [indexed].slice(0, limit) : [];
}

async function queryInfrai(body: VectorQuery): Promise<unknown> {
  const baseUrl = process.env.INFRAI_BASE_URL;
  const apiKey = process.env.INFRAI_API_KEY;
  if (!baseUrl) throw new Error("INFRAI_BASE_URL is required");
  if (!apiKey) throw new Error("INFRAI_API_KEY is required");

  const maxAttempts = 3;
  for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), 800);
    try {
      const response = await fetch(`${baseUrl}/vector/query`, {
        method: "POST",
        headers: {
          Authorization: `Bearer ${apiKey}`,
          "Content-Type": "application/json"
        },
        body: JSON.stringify(body),
        signal: controller.signal
      });

      if (response.status === 429 && attempt < maxAttempts) {
        const retryAfterSeconds = Number(response.headers.get("retry-after") ?? "0");
        await wait(Math.max(retryAfterSeconds * 1000, 100 * 2 ** (attempt - 1)));
        continue;
      }
      if (!response.ok) {
        throw new Error(`vector query rejected (${response.status}): ${await response.text()}`);
      }
      return await response.json();
    } finally {
      clearTimeout(timer);
    }
  }
  throw new Error("vector query retry budget exhausted");
}

const wait = (ms: number) =>
  new Promise<void>((resolve) => setTimeout(resolve, ms));

async function withDeadline<T>(
  task: Promise<T>,
  timeoutMs: number
): Promise<T> {
  let timer: ReturnType<typeof setTimeout> | undefined;
  const deadline = new Promise<never>((_, reject) => {
    timer = setTimeout(() => reject(new Error("stage deadline exceeded")), timeoutMs);
  });
  try {
    return await Promise.race([task, deadline]);
  } finally {
    if (timer) clearTimeout(timer);
  }
}

async function fetchFreshWithRetry(maxAttempts: number): Promise<Evidence> {
  for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
    try {
      return await fetchWatchedPage();
    } catch (error) {
      if (!(error instanceof RateLimitError) || attempt === maxAttempts) throw error;
      const exponentialMs = 10 * 2 ** (attempt - 1);
      await wait(Math.max(error.retryAfterMs, exponentialMs));
    }
  }
  throw new Error("retry budget exhausted");
}

async function retrieveCandidate(): Promise<RetrievalResult> {
  const tenantId = "tenant_northwind";
  const access: AccessScope = "recruiter";
  const fallback = await queryIndex(tenantId, access, 5);

  try {
    const fresh = await withDeadline(fetchFreshWithRetry(3), 250);
    return {
      state: "fresh",
      evidence: [fresh],
      attempts: liveAttempts,
      reason: fresh.digest === indexed.digest ? "unchanged" : "page changed"
    };
  } catch (error) {
    return fallback.length > 0
      ? { state: "stale", evidence: fallback, attempts: liveAttempts }
      : {
          state: "unavailable",
          evidence: [],
          attempts: liveAttempts,
          reason: error instanceof Error ? error.message : "unknown error"
        };
  }
}

async function main(): Promise<void> {
  const recovery = await retrieveCandidate();
  const remote = await queryInfrai({
    collection: "tenant_northwind_candidates",
    query: "candidate with invoicing migration experience",
    limit: 5
  });
  console.log(JSON.stringify({ recovery, remote }, null, 2));
}

main().catch((error: unknown) => {
  console.error(error instanceof Error ? error.message : error);
  process.exitCode = 1;
});

Set INFRAI_BASE_URL to the documented /v1 API base, set INFRAI_API_KEY, run the file with a TypeScript runner, and inspect the JSON. The local fault injection rate-limits the first live attempt; the second succeeds within the 250 ms stage deadline, and the recovery result is fresh with portfolio_2048_v8 as its traceable document. Change the deadline to 5 to exercise the stale path. The remote result comes from a bounded candidate query against the explicit tenant collection. The important output isn't a simulated score; it is the state, source, and attempt count that make the recovery decision reviewable, next to the provider response your adapter will map into the same contract.

When connecting the page adapter, use only the request schema published by that service rather than copying fields from this local contract. The relevant verified operation for page evidence is POST /v1/web/scrape; the example calls POST /v1/vector/query for bounded retrieval. Authentication uses Authorization: Bearer with a key read from process.env.INFRAI_API_KEY. Both calls need explicit methods, status checks, and the same bounded 429 policy shown above.

Pick the operating model, then accept its catch

The backend decision follows from who will own relevance, recovery, and source freshness. There isn't a winner for every team.

Infrai's advantage for this particular workflow is concrete: the worker can use a plain REST API without installing or babysitting a client library, while one key and one bill cover the web and vector capabilities. Its public discovery surface is self-describing, and the broader platform has 295 routes across 20 modules. That reduces integration upkeep for a solo operator shipping weekly. It does not prove better candidate relevance, so evaluate returned evidence against your own labeled recruiting queries before choosing it.

Stick with Elasticsearch when you already operate it and need its search controls. Pick Pinecone when a focused managed vector service matches the boundary you want. Choose Weaviate when its deployment choices suit your team and you are prepared to test the operational path. A staged design is not suitable when policy forbids live access to candidate pages, when every answer must reflect a single transactional snapshot, or when a custom learned ranker is the product's core differentiation. In those cases, keep ingestion offline, make snapshot completion explicit, or own the ranking stack.

The revenue-per-hour test is simple. Count the integration and recovery behavior you must maintain, then ask whether any of it improves recruiter decisions. Outsource the undifferentiated parts. Keep the evidence policy.

Ship the alert only when its evidence is reviewable

A page-change alert should identify the candidate, describe the changed evidence, link to the source, and disclose the observation time. Deduplicate on the candidate, source, and new digest so repeated retrieval does not produce repeated alerts. Preserve the prior document identifier as well as the new one; the diff is otherwise impossible to audit after another update.

Before release, test three paths: fresh evidence inside budget, a 429 followed by recovery, and a deadline that returns allowed indexed evidence. Also test the empty fallback and an access mismatch. These cases are small enough to run on every change, which fits a weekly release rhythm better than a broad end-to-end suite that nobody can diagnose.

Fast is useful. Explainable is durable.

No mystery state.

The final decision rule is equally compact: use staged retrieval when an indexed candidate result remains useful for a short, disclosed interval; use live-first retrieval when the newest page is required for the decision; use offline ingestion when policy or snapshot consistency forbids either fallback. Whatever the choice, keep collection scope, access metadata, source context, retries, timeouts, and result limits in the contract rather than scattered across adapters.

References

• https://arxiv.org/abs/2005.11401
• https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
• https://docs.pinecone.io/
• https://docs.weaviate.io/weaviate