UGC Pipeline: Script-First AI Video Generation

The AI ad platform's original generation mode was reference-first: upload a viral UGC clip, let the analyzer extract beats, pacing, and visual structure, then regenerate the same ad with a different product and avatar. That pipeline worked for swipe iterations on known winners. It did not work when a brand team showed up with a script and no reference video.

Over fifteen pull requests, we shipped an entirely new product mode — script-first UGC generation — where the editor owns the narrative and the AI generates scenes to match. The paradigm shift sounds obvious in retrospect. In practice it required rethinking every gate in the pipeline: creation UX, scene planning, preview semantics, batch orchestration, and the failure modes that only surfaced once editors stopped uploading reference footage.

Reference-first generation asks: "What did this viral video do?" Script-first generation asks: "What should this video say?" Those are different products with different review gates.

Reference-first vs script-first: two pipelines, one platform

The legacy flow treated the reference video as ground truth. An analyzer extracted scene boundaries, shot types, b-roll timing, and VO pacing. Planners and generators worked backward from that structure. Editors could tweak prompts, but the skeleton came from the reference.

Script-first inverts the contract:

The backend change was not cosmetic. Scene plans no longer inherit timing from a reference analyzer. They inherit intent from the script block the editor wrote in step two of the wizard.

interface ScriptFirstRunConfig {
  mode: 'script-first';
  script: EditorScript;           // verbatim VO + scene hints
  characterId: string;            // from avatar catalog
  productId: string;              // from product catalog
  referenceVideo?: never;         // explicitly absent
}

interface ReferenceFirstRunConfig {
  mode: 'reference-first';
  referenceVideoUrl: string;
  script?: Partial<EditorScript>; // optional overrides only
}

function resolvePlanningInput(config: RunConfig): PlannerInput {
  if (config.mode === 'script-first') {
    return {
      source: 'editor-script',
      beats: segmentScriptIntoScenes(config.script),
      character: loadCharacter(config.characterId),
      product: loadProduct(config.productId),
    };
  }
  return analyzeReference(config.referenceVideoUrl);
}

The multi-step create wizard

Script-first runs do not fit a single modal. Editors need to pick a character, confirm the product, paste or edit the script, and preview how scenes will split — all before any GPU time burns. We replaced the one-shot create form with a multi-step wizard inside a modal, plus an inline single-tool dashboard where editors name their run upfront.

The wizard steps:

1. Character — picker with larger preview tiles, loading skeletons while Drive-backed avatars hydrate, and support for uploaded photos that auto-save as reusable avatars
2. Product — catalog picker with inline product preview; photo-count requirements dropped because script-first does not mirror reference framing
3. Script — editor pastes VO copy; LLM proposes scene segmentation with editable boundaries
4. Review plan — per-scene visual previews at the script-review gate before committing to full generation
5. Confirm — named run enters the queue

Naming runs early sounds trivial. On a shared runs page with dozens of in-flight jobs, "UGC Run #847" is useless. Editors now label runs at creation time — "Q3 sunscreen testimonial v2" — and the runs list sorts by recency with clickable scene thumbnails for faster triage.

Per-scene visual previews at the script-review gate

The highest-leverage gate in script-first mode is script review: the moment between "we have a plan" and "render everything." Before this shipped, editors approved text-only scene descriptions and discovered visual mismatches only after two-minute renders completed.

We added lightweight visual previews at that gate — not full video, but start-frame candidates generated from the scene prompt and character conditioning. Each scene shows:

• Script excerpt for that beat
• Two to four candidate start frames
• One-click selection of the preferred frame
• Loading indication when swapping candidates or refreshing previews

interface ScenePreviewGate {
  sceneId: string;
  scriptExcerpt: string;
  candidates: StartFrameCandidate[];
  selectedCandidateId: string | null;
  status: 'idle' | 'generating' | 'ready' | 'error';
}

async function selectStartFrame(
  runId: string,
  sceneId: string,
  candidateId: string,
): Promise<void> {
  await api.patch(`/runs/${runId}/scenes/${sceneId}`, {
    selectedStartFrameId: candidateId,
  });
  // downstream full render uses selected frame as image conditioning anchor
}

Choosing the scene start frame from multiple candidates turned out to be more important than prompt tweaking. Generative models latch onto the first frame; giving editors explicit frame choice at review time reduced full-scene regenerations by roughly 40% in the first week of production use.

Batch approve-and-render, then fix in place

Reference-first workflows naturally serialized review: compare scene N to reference frame N, regenerate, repeat. Script-first needed a different pattern because scenes are independent once the script is locked.

The batch flow:

1. Approve once — editor confirms all scene previews at the script-review gate
2. Render all — pipeline fans out scene jobs in parallel
3. Scene review — after render, editor scans the runs page with instant-pick UX and clickable scene tiles
4. Fix any scene — rejected scenes re-enter preview or full render without restarting the entire run

The scene-review UX work — instant pick, more options, clickable scenes, faster runs page — was pure frontend orchestration on top of an idempotent scene state machine. Each scene tracks its own preview → approved → rendering → review → done lifecycle. Fixing scene four never invalidates scene two.

Uploaded-photo runs: face drift and rushed voice

Script-first mode increased uploaded-photo usage. Editors pick a custom headshot instead of a catalog avatar, paste a script, and expect a polished UGC testimonial. Two bugs dominated early feedback:

• Presenter face drift — the generated subject slowly morphed away from the uploaded photo across scenes
• Rushed voice — TTS pacing compressed to fit scene duration, producing unnatural delivery on longer script beats

Face drift was an architectural issue, not a prompt issue. Uploaded photos entered the pipeline with weaker conditioning than catalog avatars, which had tuned reference weights and anti-drift negative prompts. The fix applied catalog-grade persona anchoring to uploaded-photo runs:

function buildPersonaAnchor(source: CharacterSource): PersonaAnchor {
  if (source.type === 'uploaded-photo') {
    return {
      referenceImageUrl: source.photoUrl,
      conditioningStrength: 0.82,  // higher than default catalog weight
      lockFace: true,
      antiDriftNegatives: UPLOADED_PHOTO_NEGATIVES,
      persistAsAvatar: true,       // auto-save to Drive catalog
    };
  }
  return loadCatalogAnchor(source.characterId);
}

Rushed voice required separating script pacing from scene duration. Script-first plans derive scene length from VO word count, not reference timing. When a beat ran long, the naive fix sped up ElevenLabs output. The production fix runs pacing normalization before TTS — split beats, adjust scene boundaries, or apply pitch-preserving FFmpeg compression only when copy is legally fixed.

Both fixes shipped as pipeline-level changes, not wizard tweaks. That distinction matters: script-first editors never see "conditioning strength" sliders. They see consistent faces and natural VO.

State model: what changed under the hood

Script-first runs carry a different manifest shape than reference-first runs. The critical fields:

interface ScriptFirstManifest {
  runId: string;
  name: string;                    // editor-provided at create time
  mode: 'script-first';
  scriptHash: string;
  scenes: Array<{
    id: string;
    scriptBeat: string;
    selectedStartFrameId: string;
    previewStatus: ScenePreviewStatus;
    renderStatus: SceneRenderStatus;
    approvedAt?: ISO8601;
  }>;
  batchApprovedAt?: ISO8601;       // gate before parallel render
}

The script-review gate writes selectedStartFrameId per scene. Batch approval sets batchApprovedAt, which unlocks the render fan-out worker. Scene-level fixes after render only mutate the affected scene row — no full manifest version bump unless the script itself changes.

Metrics from the first production month

What I'd do differently

I would ship the script-review gate before the full wizard. Early internal testers burned GPU credits on full renders because we had script input but no preview gate — the same mistake reference-first made years earlier, just with different inputs.

I would also define the batch state machine in a shared package on day one. Scene review UX and the runs page evolved in parallel PRs; a formal scene lifecycle enum would have prevented two slightly different status strings from reaching production.

The broader lesson

Script-first UGC is not "reference-first without a reference." It is a different product with different gates: editor script as source of truth, preview before render, batch approval with per-scene repair, and persona anchoring tuned for uploaded photos. The fifteen PRs that built it look like feature work in the commit log. Architecturally, they were a pipeline fork that shares render infrastructure but not planning semantics.

If you are adding a new generation mode to an existing video platform, ask where ground truth lives before you copy the old review UI. The answer determines every gate downstream.