Redesigning Bulk Intake: Cascading Dropdowns, Taxonomy Validation, and a Week of Polish

The platform I work on generates training data through AI agents. Operators submit bulk batches — dozens or hundreds of runs at once — specifying which sector and occupation each task should target and what scenario the agent should produce. The bulk intake form is the front door to the entire data generation pipeline. When it is wrong, bad taxonomy slugs slip into production runs, scenarios never reach the agent prompt, and operators waste time fighting a UI that does not match how they actually think about work.

Over one sprint I redesigned bulk intake from a checkbox taxonomy grid into a compact row-entry table with cascading dropdowns and per-row scenario input, hardened the API with server-side taxonomy validation, and threaded scenarios through to the pipeline enqueue path. Six smaller polish PRs shipped in the same window — filter consolidation, stale cache fixes, step filter mapping, label cleanup, sector display, and a GitHub Actions Node.js upgrade. This post walks through the bulk intake redesign first, then the companion fixes that made the admin dashboard feel finished.

The old design: combinatorial explosion

The previous bulk intake UI presented a checkbox grid of sectors. Selecting a sector caused the system to generate every occupation combination under it automatically. There was no per-row control. Occupation and scenario were optional fields buried in the expansion logic, not surfaced to the operator.

That model had three problems. First, operators could not describe what the AI agent should generate — scenario was not a first-class input. Second, the combinatorial expansion created runs the operator did not intend: checking "Healthcare" might enqueue thirty occupations when they only needed two specific ones. Third, the API accepted any string for sector and occupation slugs. A typo in a slug passed client-side validation silently and produced tasks that failed downstream in the pipeline with opaque taxonomy errors.

Bulk intake is not a form problem. It is a contract problem between operators, the taxonomy registry, and the data generation pipeline.

The new design: row-entry with cascading dropdowns

I replaced the checkbox grid with a compact table where each row is one intended run. Every row has three required fields: sector, occupation, and scenario. Sector and occupation are cascading dropdowns — selecting a sector filters the occupation list to only valid children from the vendored taxonomy registry. The scenario field is a textarea where the operator describes what the AI agent should generate for that specific row.

Row controls let operators add, remove, and duplicate rows. Duplicating is the common path when an operator wants the same sector/occupation pair with a different scenario — one click copies the taxonomy selections and clears the scenario for editing. Submit stays disabled until every row passes client-side validation: non-empty occupation, non-empty scenario, and a sector/occupation pair that exists in the registry.

interface BulkIntakeRow {
  id: string;
  sector: string;
  occupation: string;
  scenario: string;
}

function occupationsForSector(
  sectorSlug: string,
  registry: TaxonomyRegistry,
): OccupationOption[] {
  const sector = registry.sectors.find((s) => s.slug === sectorSlug);
  if (!sector) return [];
  return sector.occupations.map((occ) => ({
    slug: occ.slug,
    label: occ.displayName,
  }));
}

function validateRow(row: BulkIntakeRow, registry: TaxonomyRegistry): string[] {
  const errors: string[] = [];
  if (!row.sector) errors.push("Sector is required");
  if (!row.occupation?.trim()) errors.push("Occupation is required");
  if (!row.scenario?.trim()) errors.push("Scenario is required");
  if (row.sector && !registry.hasSector(row.sector)) {
    errors.push(`Unknown sector: ${row.sector}`);
  }
  if (row.sector && row.occupation && !registry.hasOccupation(row.sector, row.occupation)) {
    errors.push(`Unknown occupation for sector`);
  }
  return errors;
}

The cascading dropdown pattern is straightforward on the client but it eliminates an entire class of user error. Operators cannot select an occupation that does not belong to the chosen sector because the occupation dropdown simply does not offer invalid options.

Server-side taxonomy validation

Client-side cascading dropdowns are not sufficient. A crafted request could still POST invalid slugs to POST /api/batches. I added server-side validation that checks every sector and occupation slug against the same vendored taxonomy registry before any runs are created.

const BulkRunCellSchema = z.object({
  sector: z.string().min(1),
  occupation: z.string().min(1),
  scenario: z.string().min(1),
});

async function validateBulkCells(
  cells: z.infer<typeof BulkRunCellSchema>[],
  registry: TaxonomyRegistry,
): Promise<void> {
  for (const [index, cell] of cells.entries()) {
    if (!registry.hasSector(cell.sector)) {
      throw new ValidationError(
        `Row ${index + 1}: unknown sector slug "${cell.sector}"`,
      );
    }
    if (!registry.hasOccupation(cell.sector, cell.occupation)) {
      throw new ValidationError(
        `Row ${index + 1}: occupation "${cell.occupation}" ` +
        `is not valid for sector "${cell.sector}"`,
      );
    }
  }
}

Previously, bad slugs passed silently and surfaced as pipeline failures hours later. Now the API returns a 400 with a row-indexed error message at submission time. The registry is vendored — a checked-in JSON artifact versioned with the application — so validation is deterministic across environments and does not depend on a live taxonomy service.

Scenario threading through the pipeline

Making scenario required on the schema was only half the work. The scenario also had to reach the AI agent prompt. I updated BatchManifestService.createBatch to pass each row's scenario through createRun and into the enqueue metadata that the data generation pipeline consumes.

async createBatch(input: CreateBatchInput): Promise<Batch> {
  await validateBulkCells(input.cells, this.taxonomyRegistry);

  const batch = await this.db.batch.create({ data: { /* ... */ } });

  for (const cell of input.cells) {
    const run = await this.createRun({
      batchId: batch.id,
      sector: cell.sector,
      occupation: cell.occupation,
      scenario: cell.scenario,
    });

    await this.enqueueService.enqueue({
      runId: run.id,
      metadata: {
        sector: cell.sector,
        occupation: cell.occupation,
        scenario: cell.scenario,
      },
    });
  }

  return batch;
}

Before this change, the agent prompt received sector and occupation context but not the operator's intent. Operators were describing scenarios in Slack or spreadsheets because the product had no place to put them. Threading scenario through enqueue metadata closed that gap — each run now carries the operator's description from intake through to generation.

Dedup compatibility

One wrinkle: deduplication callers submit bulk cells with nullable occupation because dedup matching keys off sector alone in some code paths. I used an inline type for those callers instead of the shared BulkRunCell type, keeping the strict schema on the intake path without breaking the dedup service's narrower contract.

Companion polish PRs from the same sprint

The bulk intake redesign was the headline, but six smaller PRs shipped alongside it. Each one removed friction operators had been working around.

The stale cache fix and step filter mapping are worth highlighting together. Both were invisible bugs — the UI rendered plausible but wrong state. An operator filtering by step would miss gate-approved tasks entirely because they appeared under "Finished." Reopening a completed task from the list would flash old progress data from a client-side cache replay. Neither threw an error. Both made the dashboard feel unreliable.

What changed in practice

The bulk intake redesign landed at +474 / −288 lines across eight commits. The UI went from a combinatorial checkbox grid to an intentional row-entry workflow. The API went from accepting any slug to rejecting invalid taxonomy at the front door. Scenarios flow from operator input through to the agent prompt instead of living in side channels.

The companion polish PRs were smaller individually — three lines for the stale cache fix, fifty for the step filter mapping, fourteen for the label rename — but they addressed the kind of friction that erodes trust in an admin dashboard. Operators stopped seeing truncated sector slugs, stopped hunting for tasks that the step filter hid, and stopped wondering why a completed task looked like it was still running.

The through-line across all seven PRs is the same principle: make the contract explicit at every boundary. Cascading dropdowns enforce valid sector/occupation pairs in the UI. Server-side validation enforces them in the API. Scenario threading enforces operator intent in the pipeline. Filter mapping and cache fixes enforce that what the dashboard shows matches what the system actually did. Bulk intake is the front door, but the whole admin experience has to be trustworthy for operators to move fast without breaking production.