3 Candidate Content Policy Checks Before Review — Why LLM Moderation False Positives Happen

Short answer: LLM moderation false positives happen when model signals are treated as policy verdicts; for user-generated candidate content, allow clear cases, send uncertainty to a review queue, and block only narrow high-confidence matches.

For an e-commerce marketplace scoring candidates against a job rubric, the default should be allow, review, block, in that order of preference. The model supplies evidence. A versioned policy function owns the route. One model score should never silently become an employment decision.

This is a decision note, not a claim that one set of numbers works everywhere. Start with an explicit three-route contract, replay labeled tenant data through it, and promote threshold changes only when both error rates and review workload remain acceptable.

How should US and EU marketplaces route LLM moderation false positives for user-generated candidate content?

False positives happen because a moderation label answers a narrower question than the application does. A classifier may identify a phrase associated with harassment, self-harm, sexual content, or violence, while the hiring workflow needs to know whether that phrase violates a specific rule in context. A candidate describing prior trust-and-safety work can quote the exact language they were paid to investigate. Keyword-like evidence is present; prohibited intent may not be.

Consider a hypothetical application for a marketplace safety analyst. The candidate pastes a short redacted example of an abusive seller message, explains which phrase triggered escalation, and describes how they protected the buyer. A classifier assigns the quoted phrase the strongest risk score in the document. A hard gate rejects the submission, even though the surrounding answer is evidence for the job rubric. An allow-only flow has the opposite weakness because a genuinely abusive message aimed at a recruiter would proceed. The review route preserves both possibilities: the event records the triggering category and policy version, the reviewer sees the permitted context, and the rubric scorer receives content only after the moderation state is resolved. Now split that workflow across tenants. One tenant may permit redacted case studies; another may forbid pasted customer messages for confidentiality reasons. The model signal hasn't changed, but the applicable policy has. This is why the route belongs to a tenant-aware policy layer rather than a generic score cutoff.

Context gets thinner as systems add glue. A long application may be truncated before classification. A multilingual answer may be normalized. A rubric scorer may receive a moderation result without the text span that triggered it. Then a continuous score gets rounded into a Boolean because a downstream schema has only approved: true | false. Each step discards information, and the last step looks deceptively certain.

Don't repair that loss by picking a more confident-sounding threshold. Keep the raw category scores, the policy version, the model identifier, the evaluated text hash, and the resulting route together. Store only what your retention rules permit. The useful invariant is that an operator can reconstruct why a decision entered the queue without reading tea leaves from a single blocked flag.

US and EU traffic also shouldn't be reduced to two magic constants. Region can select a reviewed policy bundle, but geography alone doesn't settle which employment, privacy, platform, or record-retention obligations apply. Those depend on the marketplace, tenant, candidate location, role, and actual use of the output. I'm not sure any static country switch can stay correct without policy and legal owners reviewing the surrounding workflow; the engineering answer is to make that ownership and version history visible.

Small distinction. Big consequence.

Criterion one: measure routing errors against the job rubric

The denominator matters. “False-positive rate” can mean flagged items that reviewers clear, or allowed items that should never have been flagged, or all submissions that were unnecessarily interrupted. Pick one definition in the evaluation spec and keep the confusion matrix beside it. Otherwise two teams can report improvement while measuring different populations.

Build the test set from candidate-generated content that resembles production inputs, segmented by tenant, locale, role family, and content length. Preserve hard cases: quoted abuse from a safety role, security terminology from a penetration tester, medical language from a benefits specialist, and ordinary words whose meaning changes by locale. Synthetic examples can probe a boundary, but they don't establish the real distribution.

The scoring rubric and the safety policy must remain separate artifacts. Moderation decides whether content can proceed, needs review, or must stop under a stated rule. Rubric scoring evaluates job evidence. If the same prompt performs both jobs, a safety term can leak into candidate ranking even after a reviewer clears the content. That's a nasty coupling — and it is hard to audit.

Use a shadow run before changing enforcement. Replay a fixed, labeled set through the candidate policy, compare it with the current policy, then inspect disagreements by segment. A global average can hide a sharp regression for one language or one tenant. Benchmarks need slices.

Criterion two: expose cost per tenant, not just per request

Review is safer than an aggressive block threshold only while the queue remains operable. That makes cost visibility part of policy quality. For every routed item, attribute model input units, model output units, retry count, review minutes, and storage or log volume to a tenant identifier and policy version. Keep money conversion outside the routing function because provider rates and internal labor assumptions change on a different schedule.

The dashboard should join quality and cost rather than displaying two unrelated charts. For each tenant and policy version, show submission count, allow/review/block counts, reviewer overturns, queue age, model units, and estimated review effort. A tenant producing long multilingual applications may have an acceptable model error rate but a costly review tail. Another may have low volume and high disagreement. One blended average conceals both.

This is also where config bloat starts. Resist a threshold per tenant, category, locale, role, and model unless the data demonstrates a stable difference. The Cartesian product becomes impossible to test. Prefer a small number of named policy bundles with explicit owners and effective dates; attach tenants to bundles, then demand evaluation evidence before adding another.

No mystery knobs.

A typed policy boundary with three explicit outcomes

The useful interface is boring: normalized evidence goes in, a route plus reasons comes out. The example thresholds below are illustrative configuration, not recommended universal values. Their purpose is to show separation: review begins at 0.55, while blocking requires 0.92 and an explicit category allowlist. Change them only through a versioned evaluation.

type Route = "allow" | "review" | "block";

type Signal = {
  category: string;
  score: number;
};

type Policy = {
  version: string;
  reviewAt: number;
  blockAt: number;
  blockableCategories: ReadonlySet<string>;
};

type Decision = {
  route: Route;
  policyVersion: string;
  reasons: string[];
};

function routeContent(signals: Signal[], policy: Policy): Decision {
  const strongest = [...signals].sort((a, b) => b.score - a.score)[0];

  if (!strongest || strongest.score < policy.reviewAt) {
    return { route: "allow", policyVersion: policy.version, reasons: [] };
  }

  const canBlock =
    strongest.score >= policy.blockAt &&
    policy.blockableCategories.has(strongest.category);

  return {
    route: canBlock ? "block" : "review",
    policyVersion: policy.version,
    reasons: [`${strongest.category}:${strongest.score.toFixed(3)}`],
  };
}

Keep tenant billing metadata out of routeContent; pass it through an event envelope so policy tests stay deterministic. An event can contain tenantId, submissionId, policyVersion, unit counts, latency, and the route. The candidate text itself doesn't need to be copied into every cost event. This boundary makes it possible to swap a classifier or gateway without rewriting enforcement semantics, while a stable event schema preserves per-tenant comparisons.

Test boundaries, not happy-path screenshots. At minimum, cover values immediately below and at both thresholds, an empty signal list, a high score in a non-blockable category, ties, and invalid scores rejected during normalization. Then replay the labeled corpus. Unit tests prove the switch does what it says; they can't prove that the policy is fair or useful.

When is a binary gate or full manual review the better choice?

The three-route design has a catch: it adds a queue, an operator interface, escalation rules, access controls, and feedback handling. It is not suitable when nobody is accountable for review latency. In that case, “review” is merely a delayed block. For a narrow input with deterministic prohibited values, stick with validation rules or a binary gate; an LLM adds ambiguity without buying context.

Full manual review is the better runner-up when volume is low, candidate impact is high, labeled examples are scarce, or policy is changing too quickly to evaluate automation. It costs more operator time, but it creates adjudicated examples and exposes disagreements before code freezes them into a threshold. Move toward assisted triage only after reviewers can state the rule consistently.

At the other extreme, an allow-first flow can fit low-risk fields where users can edit after publication and reports are handled quickly. It should not be borrowed for irreversible candidate rejection. The blast radius is different.

No vendor choice removes these trade-offs. A reranker can order review candidates, and a self-hosted gateway can centralize model calls, yet neither defines the marketplace's policy or owns the hiring consequence. Evaluate such components on time-to-first-call, preservation of evidence, exportable usage metadata, and the amount of glue required to keep policy separate from model output. If a tool can't expose tenant attribution without parsing logs after the fact, its clean demo is hiding operational work.

The final decision rule is plain: automate routing only to the level you can evaluate, explain, staff, and meter per tenant. Keep uncertain content reversible. Keep rubric scoring downstream of a cleared moderation state. Block only on narrow rules whose errors have been measured against the people who will feel them.

References

• https://docs.cohere.com/docs/rerank-overview
• https://github.com/BerriAI/litellm