Vision-in-the-Loop: When the AI Rewrites Its Own Prompts from the Generated Frame

On the AI video ad platform I work on, every scene goes through the same painful loop: write a prompt, send it to an AI video model provider, wait two minutes, open the result, squint at the frame, and decide what went wrong. Camera too wide. Product missing from the hero shot. Color palette drifted warm when the brand brief says cool neutrals. Avatar looks like a different person than scene three.

That loop was manual, slow, and expensive. Each regeneration burns GPU credits. Operators were becoming prompt engineers by accident — and still missing subtle failures until stitch time, when fixing scene four means re-rendering everything downstream.

The insight behind vision-in-the-loop prompt authoring is simple: the model that wrote the prompt can also look at its own output and rewrite the prompt with surgical fixes. Not a full replan — a per-scene correction grounded in the actual generated frame, not the operator's memory of what they hoped would appear.

The manual loop we were trying to kill

Before this work shipped, the swipe iteration flow looked like this:

1. Plan — Claude generates a scene-by-scene script with visual prompts
2. Generate — each scene renders independently through an AI video model provider
3. Review — operator opens the portal, compares frames to the reference ad
4. Rewrite — operator edits prompts in a text field, often guessing at what the model misread
5. Regenerate — repeat until acceptable or budget exhausted

Steps three and four are where throughput dies. An experienced operator can spot "product not visible" in three seconds, but translating that into prompt language — "medium close-up, product centered in lower third, shallow depth of field" — takes another minute per scene. Multiply by twelve scenes and three swipe iterations, and a single ad creative consumes an hour of human attention that should be spent on brand strategy, not frame inspection.

The generated frame is ground truth. The original prompt is a hypothesis. Vision-in-the-loop closes the gap between them automatically.

Vision-in-the-loop: iteration 3 architecture

The third swipe iteration introduced a per-scene feedback loop that runs immediately after the first frame of each scene is generated — before the operator ever opens the portal.

The vision model receives the generated frame alongside the scene's intent metadata — shot type, product placement rules, palette constraints, and whether this is a person scene or product B-roll. It returns a structured critique, not free-form prose. That structure matters: downstream rewrite logic keys off issue categories like framing_too_wide, product_absent, palette_drift, and identity_mismatch.

interface VisionCritique {
  sceneId: string;
  frameUrl: string;
  issues: Array<{
    category: VisionIssueCategory;
    severity: 'blocker' | 'warning' | 'nit';
    evidence: string;       // what the model sees in the frame
    suggestedFix: string;   // natural-language correction hint
  }>;
  passThreshold: boolean;   // true if no blockers
}

async function visionInLoopRewrite(
  scene: ScenePlan,
  generatedFrame: Buffer,
): Promise<ScenePromptRevision | null> {
  const critique = await visionCritique({
    frame: generatedFrame,
    originalPrompt: scene.visualPrompt,
    intent: scene.intent,
    brandRules: scene.brandConstraints,
  });

  if (critique.passThreshold) return null;

  const blockers = critique.issues.filter(i => i.severity === 'blocker');
  if (blockers.length === 0) return null;

  return promptRewriter.revise({
    original: scene.visualPrompt,
    issues: blockers,
    preserveLocked: scene.lockedElements, // persona anchor, product lock, etc.
  });
}

The rewrite step is constrained: it may not change locked elements (avatar reference, product SKU imagery, mandated taglines). It only adjusts the visual prompt fields that the vision critique flagged. This prevents the common failure mode where an aggressive rewrite "fixes" the framing but drops the brand voice or swaps the protagonist.

What the vision model actually catches

In production testing across roughly 200 scene generations, the vision critique surfaced issues that text-only plan QA missed entirely:

• Framing drift — wide establishing shot when the plan called for medium close-up on the spokesperson
• Product absence — hero product not visible despite explicit product-lock instructions in the prompt
• Palette mismatch — warm golden-hour tones on a brand that specifies cool clinical whites
• Background clutter — competing visual elements that dilute the focal subject
• Motion mismatch — static hold when the reference scene had subtle camera push-in

Each issue maps to a prompt delta. "Framing too wide" becomes tighter focal length language and explicit subject placement. "Product absent" triggers a product-lock reinforcement clause and shot-type downgrade to guarantee visibility. The rewrite is incremental — we append and refine, not replace wholesale.

Motion-transfer defaults for person scenes

Vision-in-the-loop solves prompt accuracy. A parallel iteration — swipe iteration 2 — solved a different problem: person scenes that felt dead on arrival.

Generic text-to-video generation treats every scene the same. For product B-roll, that works — a slow pan across a bottle on marble is fine with standard generation. For person scenes, the reference ad almost always has subtle body motion, micro-expressions, and natural idle movement that pure text-to-video renders as uncanny mannequin holds.

The fix: make motion-transfer the default generation mode for any scene classified as a person scene. Motion-transfer takes a reference clip (from the source ad or a prior good take) and transfers the subject's motion onto the generated avatar, preserving liveliness while swapping identity.

function resolveGenerationMode(scene: ScenePlan): GenerationMode {
  if (scene.classification === 'person') {
    return {
      mode: 'motion-transfer',
      referenceClip: scene.referenceMotionClip ?? scene.referenceKeyframe,
      holdPolicy: 'lively', // shorter static holds, subtle idle motion
      brollCentering: 'person', // B-roll cuts stay person-centered, not product
    };
  }
  return { mode: 'standard', holdPolicy: 'default' };
}

Two supporting changes shipped alongside the default:

• Livelier holds — reduced minimum hold duration on person scenes so the model doesn't freeze the subject into a portrait pose for three seconds
• Person-centered B-roll — when a person scene cuts to supplementary footage, the B-roll framing centers the human subject rather than defaulting to product hero shots that break narrative continuity

The impact was immediate: first-pass acceptance rate on person scenes climbed because the output moved like the reference, not because the prompt was more poetic.

Avatar persona anchoring: stopping identity drift

Even with better prompts and motion-transfer, multi-scene ads had a persistent quality problem: the avatar looked like one person in scene two and a cousin in scene seven. Generative models don't maintain identity across independent renders — each scene is a fresh diffusion run with no memory of prior frames.

The persona anchoring fix injects the same reference image — the approved avatar headshot or prior best frame — into every person-scene generation call, not just the first one.

function buildPersonScenePayload(
  scene: ScenePlan,
  personaAnchor: PersonaAnchor,
): VideoGenerationRequest {
  return {
    prompt: scene.visualPrompt,
    imageConditioning: {
      primary: personaAnchor.referenceImageUrl,
      weight: personaAnchor.conditioningStrength, // typically 0.7–0.85
      lockFace: true,
    },
    motionTransfer: scene.motionReference,
    negativePrompt: personaAnchor.antiDriftNegatives,
  };
}

Before anchoring, operators reported "face drift" on roughly 30% of multi-scene runs. After anchoring on every person scene — not just scene one — drift dropped to single digits. The remaining failures were usually extreme angle changes (profile shots) where conditioning weight needed per-scene tuning, not absent anchoring.

Vision-in-the-loop and persona anchoring compose cleanly: if the vision critique flags identity_mismatch, the rewrite step increases conditioning weight before regeneration rather than rewriting the entire character description.

Pitch-preserving VO time-compression

A separate but related pacing fix shipped in the same arc: verbatim voiceover scripts that ran longer than the reference ad's runtime.

Advertisers often provide exact copy — legal claims, mandated disclaimers, taglines that cannot be paraphrased. When the script exceeds the reference duration, the naive fix is TTS at 1.3× speed, which sounds rushed and chipmunk-adjacent. The production fix applies pitch-preserving time-compression via FFmpeg's atempo filter chain, keeping the speaker's natural pitch while fitting the audio bed to the reference length.

# Pitch-preserving compression: 45s VO -> 38s target
ffmpeg -i vo_raw.wav \
  -filter:a "atempo=1.05,atempo=1.05,atempo=1.05" \
  -y vo_compressed.wav
# atempo caps at 2.0 per filter instance; chain for ratios > 2.0

This runs after TTS render and before stitch, only when the pacing module detects over-budget duration on a verbatim (non-rewritable) script. Rewritable scripts still go through Claude pacing auto-fit first — compression is the fallback when the words are fixed.

The feedback loop as a system pattern

Vision-in-the-loop is not just a feature — it's a reusable pattern for any generative pipeline where the output is inspectable:

1. Generate with current parameters
2. Extract a inspectable artifact (frame, audio segment, JSON output)
3. Critique with a multimodal model against intent metadata
4. Revise inputs surgically, respecting locked constraints
5. Regenerate once (with a per-scene cap to control cost)

The cost tradeoff is one extra vision API call per scene — cheap relative to a full video regeneration. We cap at one automatic rewrite per scene per swipe iteration; if the second render still fails vision critique, it surfaces to the operator with the structured issue list attached, so their manual edit starts from diagnosis rather than guesswork.

What I'd do differently

I'd instrument the vision critique categories earlier. We added logging after the first week and immediately saw that product_absent clustered on a specific model provider's aspect-ratio handling — a vendor issue masquerading as a prompt problem. Earlier telemetry would have shortened the debugging cycle.

I'd also batch vision critiques where scenes share a generation queue, rather than serializing them. The critique is I/O-bound; parallelizing across scenes in the same job saves 30–40 seconds on a twelve-scene run without increasing regeneration cost.

How vision-in-the-loop fits the swipe iteration arc

The feature shipped across three swipe iterations, each building on the last:

Iteration 3 only worked because iterations 1 and 2 had already established the data model for scene prompts, locked elements, and generation modes. Vision-in-the-loop is a feedback mechanism layered on a stable generation contract — not a standalone feature.

The broader lesson: in generative systems, the model's output is the best input for the next iteration. Closing that loop inside the pipeline — instead of exporting it to a human reviewer — is how you scale from demo to production without hiring an army of prompt engineers.