Trainer Gate QC: Integrating Quality Checks into a Multi-Stage Review Pipeline

AI training data pipelines do not end at model output. Human reviewers — trainers — validate, correct, and approve data before it enters the training set. The trainer gate is that checkpoint. Integrating automated QC into it means checks run at the right moment in the review lifecycle: not so early that reviewers cannot fix issues, not so late that bad data ships.

On the enterprise workflow platform, I shipped QC orchestration into the trainer review gate across a cluster of PRs — from the core integration (AGT-770) through spinner scoping, manifest transitions, claim lifecycle edge cases, and force-pass guardrails. This post covers what that integration looks like and why the edge cases matter more than the happy path.

Multi-stage pipeline UIs look simple in diagrams. They are hard because every stage has its own claim, timeout, and release semantics — and they interact.

The trainer gate in context

A typical task on the platform moves through stages:

1. Intake — task created, assigned to queue
2. Model output — AI generates initial training data
3. Trainer gate — human reviewer claims, edits, approves
4. QC orchestration — automated checks validate the reviewer's output
5. Release — approved data enters the training corpus

AGT-770: Integrating QC orchestration

The core feature wired the existing QC orchestration service into the trainer gate submission flow. Before AGT-770, QC ran as a separate post-release step — too late to catch issues reviewers could fix. After AGT-770, submitting a trainer review triggers QC automatically:

async function onTrainerSubmit(
  taskId: string,
  review: TrainerReview,
  manifest: TaskManifest,
): Promise<SubmitResult> {
  // Advance manifest to submitted
  await manifest.transition('trainer_gate', 'submitted', {
    reviewerId: review.reviewerId,
    submittedAt: Date.now(),
  });

  // Dispatch QC orchestration
  const qcJob = await qcOrchestrator.dispatch({
    taskId,
    stage: 'trainer_gate',
    payload: review.output,
    checks: manifest.qcConfig.trainerGateChecks,
  });

  return { status: 'qc_pending', qcJobId: qcJob.id };
}

QC orchestration runs configured checks — format validation, golden data comparison, LLM adjudication — asynchronously via Cloud Tasks workers. The trainer sees results in the Task Details UI when checks complete.

Manifest state transitions

AGT-865 ensured manifest advances to trainer_gate/pending when model output finalizes — the handoff point where tasks become claimable by trainers. Without this transition, tasks sat in limbo between model completion and trainer visibility.

async function onModelOutputFinalize(
  taskId: string,
  output: ModelOutput,
): Promise<void> {
  const manifest = await manifestStore.get(taskId);
  await manifest.transition('model_output', 'finalized', { outputHash: hash(output) });
  await manifest.transition('trainer_gate', 'pending', {
    claimableAt: Date.now(),
    autoSeedStatus: manifest.autoSeed?.status ?? null,
  });
  await eventBus.emit('task.trainer_gate.pending', { taskId });
}

Spinner scoping: progress only when QC runs

AGT-1082 fixed a UX bug where the auto-seed status spinner appeared for every manifest state change — not just when Auto-QC was actually in progress. Reviewers saw spinning indicators during unrelated transitions (claim, release, draft save) and could not tell whether QC was running.

function shouldShowQcSpinner(manifest: TaskManifest): boolean {
  const stage = manifest.currentStage;
  const status = manifest.currentStatus;
  // Only show spinner when Auto-QC is actively running
  return stage === 'trainer_gate'
    && status === 'submitted'
    && manifest.qcJob?.state === 'in_progress';
}

// Before AGT-1082: spinner on ANY trainer_gate status change
// After AGT-1082: spinner ONLY on submitted + qc in_progress

Spinner scoping sounds trivial. In a pipeline where status changes fire every few seconds (claim timeout checks, webhook callbacks, draft auto-save), unscoped spinners erode trust in the UI — reviewers stop believing any spinner means something is actually happening.

Claim lifecycle edge cases

The hardest part of multi-stage pipeline UIs is not the happy path. It is what happens when things go sideways:

Claim timeout and release

When a trainer claims a task but does not submit within the timeout window, the claim expires. AGT-1103 (in progress) addresses draft and manifest reset at release time — both manual release and auto-timeout:

async function onClaimRelease(
  taskId: string,
  reason: 'manual' | 'timeout',
  manifest: TaskManifest,
): Promise<void> {
  // Reset in-progress draft
  await draftStore.delete(taskId, manifest.claimedBy);

  // Reset manifest to pending (reclaimable)
  await manifest.transition('trainer_gate', 'pending', {
    releasedAt: Date.now(),
    releaseReason: reason,
    previousClaimer: manifest.claimedBy,
  });

  // Preserve submission history for audit
  await auditLog.record('claim.released', { taskId, reason });
}

Released tasks remain in My Tasks

AGT-1145 fixes a redirect bug: after releasing a task, it disappeared from the reviewer's "My Tasks" list even though the release was intentional (not a completion). Reviewers lost visibility into tasks they had started but chose to return to the queue.

Force-pass with incomplete golden data

AGT-898 blocks force-pass QC when the golden data form is incomplete. Force-pass is an admin escape hatch — bypass QC checks when they are known to be wrong for a specific task. Without the guard, admins could force-pass tasks missing required golden data fields, shipping incomplete training examples.

Admin-only pipeline observability

AGT-886 restricted Task Details pipeline and events tabs to admin users. Trainers saw raw manifest JSON and webhook event logs that created confusion ("why is my task stuck?") without providing actionable information. Admins need pipeline observability for debugging; trainers need a simplified status view.

function TaskDetailsTabs({ task, user }: Props) {
  const tabs = [
    { id: 'review', label: 'Review', visible: true },
    { id: 'qc-results', label: 'QC Results', visible: true },
    { id: 'pipeline', label: 'Pipeline', visible: user.role === 'admin' },
    { id: 'events', label: 'Events', visible: user.role === 'admin' },
  ];
  return <TabBar tabs={tabs.filter(t => t.visible)} />;
}

QC orchestration interaction model

The trainer gate integration follows a request-async-respond pattern:

1. Trainer submits review → manifest transitions to submitted
2. QC orchestrator dispatches checks to configured agents
3. Cloud Tasks worker polls agent webhooks for results
4. On pass → manifest transitions to qc_passed, task proceeds to release
5. On fail → manifest transitions to qc_failed, task returns to reviewer with failure details

Reviewers never wait synchronously for QC. The spinner (scoped by AGT-1082) indicates progress; results appear when ready. This decoupling is essential — QC checks involving LLM adjudication can take 30–90 seconds.

Why edge cases dominate pipeline UI work

The happy path — claim, review, submit, QC pass, release — is straightforward to implement. It accounts for maybe 60% of task volume. The remaining 40% is edge cases:

• Trainer claims, gets distracted, claim times out
• Trainer submits, QC fails, trainer reclaims and resubmits
• Admin force-passes a task with known QC false positive
• Auto-seed generates draft data while trainer is mid-review
• Two trainers attempt to claim the same task simultaneously

Each edge case requires manifest state consistency, draft cleanup, audit trail preservation, and UI feedback that matches the actual system state. Getting any one wrong produces "ghost tasks" — tasks that appear claimed but are not, or appear released but still hold a draft lock.

What I'd do differently

I would define claim lifecycle as an explicit state machine diagram in the repo before implementing AGT-770. We discovered edge cases incrementally through production tickets rather than modeling them upfront. A formal state machine with documented transitions would have caught AGT-1145 (post-release visibility) during design.

I would also emit structured events for every manifest transition from day one, not as a follow-up (AGT-886's events tab). Pipeline observability should be built into the transition layer, not bolted onto the UI.

The pattern: integrate QC at the gate, not after release

QC orchestration existed before the trainer gate integration. The value of AGT-770 was placement — running checks when the reviewer submits, while they can still fix issues, rather than after release when bad data is already in the corpus. Spinner scoping, claim lifecycle resets, and force-pass guardrails are the unglamorous work that makes the integration trustworthy in production.