Dual Aspect-Ratio Rendering: Synchronizing 16:9 Widescreen and 9:16 Portrait AI Ads

Ad creative rarely ships in one shape. A vertical hook may work for Reels or TikTok, while a sales page, YouTube placement, or desktop embed needs widescreen. The tempting shortcut is to render 9:16 and crop it into 16:9 later. That shortcut breaks quickly: presenters lose shoulders, product callouts drift outside the frame, captions sit in the wrong safe area, and the composition starts looking like an afterthought.

In the AI ad/video platform, Hook Machine started as a 9:16-first tool because short-form hooks were the main use case. PR #56 (feat/voice-picker-and-16x9) changed the output contract: a generated hook could ship with a voice the editor could preview and with 16:9 output alongside 9:16. That forced a renderer-level decision. We could either crop after the fact, or render parallel spatial layouts against a single audio source of truth. The second approach won.

Never crop vertical video into widescreen post-hoc. Render parallel spatial coordinate trees anchored to one synthesized voiceover track.

Why crop-after-render fails

Video crops look simple when the subject sits in the exact center and there are no overlays. AI-generated ads rarely satisfy those constraints. The output may include presenter framing, product lockups, caption highlights, CTA badges, progress indicators, and scene-specific visual emphasis. A crop that preserves the face may cut the product. A crop that preserves the caption may weaken the hook's first-frame impact.

This is why the dual-ratio work connects back to Script-First UGC: once the script and scene plan are structured, layout can adapt per target surface without asking the model to regenerate the whole ad.

Multi-canvas layout engine

The renderer keeps logical elements independent from physical pixels. A scene does not say "draw the avatar at x=120." It says "draw presenter primary, captions bottom-safe, product proof secondary." The target canvas resolves those slots differently for 9:16 and 16:9.

type AspectRatio = '9:16' | '16:9';

type SceneElement =
  | { kind: 'presenter'; priority: 'primary' | 'secondary' }
  | { kind: 'caption'; placement: 'safe-bottom' | 'lower-third' }
  | { kind: 'product'; placement: 'proof-panel' | 'hero' }
  | { kind: 'cta'; placement: 'badge' | 'end-card' };

type LayoutRect = { x: number; y: number; w: number; h: number };

function resolveLayout(
  element: SceneElement,
  aspect: AspectRatio,
  canvas: { width: number; height: number },
): LayoutRect {
  if (element.kind === 'presenter' && aspect === '9:16') {
    return { x: canvas.width * 0.10, y: canvas.height * 0.12, w: canvas.width * 0.80, h: canvas.height * 0.54 };
  }

  if (element.kind === 'presenter') {
    return { x: canvas.width * 0.06, y: canvas.height * 0.10, w: canvas.width * 0.42, h: canvas.height * 0.78 };
  }

  if (element.kind === 'caption') {
    return aspect === '9:16'
      ? { x: canvas.width * 0.08, y: canvas.height * 0.74, w: canvas.width * 0.84, h: canvas.height * 0.16 }
      : { x: canvas.width * 0.10, y: canvas.height * 0.76, w: canvas.width * 0.80, h: canvas.height * 0.14 };
  }

  return aspect === '9:16'
    ? { x: canvas.width * 0.18, y: canvas.height * 0.56, w: canvas.width * 0.64, h: canvas.height * 0.14 }
    : { x: canvas.width * 0.56, y: canvas.height * 0.18, w: canvas.width * 0.34, h: canvas.height * 0.50 };
}

The code is not trying to be a design system. It is an anti-crop layer. It keeps meaning stable while pixels move. That makes testing easier too: a snapshot can assert that the product remains visible in both exports and that captions stay inside safe areas.

One audio source of truth

The audio contract matters more than it looks. If 9:16 and 16:9 renders synthesize voice separately, even tiny duration differences make caption timings, lip-sync, and cuts drift. PR #56 made voice preview part of the editor experience, but the render path still needed to treat the approved voice asset as canonical. Both aspect ratios consume the same voiceover, word timings, and trim metadata from Automated Dead-Air Trimming.

That design also keeps generation cost down. The system pays for voice synthesis once, then spends CPU to render two visual layouts. If the user changes the voice, both ratios invalidate. If the user changes only the target surface, the audio cache stays valid.

type RenderPlan = {
  jobId: string;
  voice: {
    audioPath: string;
    words: Array<{ text: string; startMs: number; endMs: number }>;
    trimMs: number;
  };
  outputs: Array<{ aspect: AspectRatio; frameDir: string; outputPath: string }>;
};

async function renderAllRatios(plan: RenderPlan) {
  await Promise.all(plan.outputs.map(output =>
    renderFrames({
      aspect: output.aspect,
      words: plan.voice.words,
      frameDir: output.frameDir,
    }),
  ));

  return muxOutputsWithSharedVoice(plan);
}

Isolating audio from frame capture

Frame capture should be visually pure. PR #58 in the same Hook Machine cluster found an inline audio element that could hide a late-starting hook and burn UI into captured frames. The dual-ratio renderer adopted the same separation: preview controls live in the interactive UI; render scenes receive only the visual tree. Audio enters later during FFmpeg mux.

That separation is what lets two visual exports share one voice track cleanly. It also keeps headless capture deterministic. The renderer should not depend on browser autoplay policies, hidden media controls, or preview component state.

FFmpeg dual export pipeline

The final mux step is deliberately boring: each aspect ratio produces frames, then FFmpeg combines those frames with the approved voiceover. The important part is that both outputs consume the same audio file and the same normalized timestamps.

ffmpeg -y \
  -framerate 30 -i frames_916/%05d.png -i voice.wav \
  -filter_complex "[0:v]setpts=PTS-STARTPTS[v916];[1:a]asetpts=PTS-STARTPTS[a916]" \
  -map "[v916]" -map "[a916]" -c:v libx264 -c:a aac output_916.mp4

ffmpeg -y \
  -framerate 30 -i frames_169/%05d.png -i voice.wav \
  -filter_complex "[0:v]setpts=PTS-STARTPTS[v169];[1:a]asetpts=PTS-STARTPTS[a169]" \
  -map "[v169]" -map "[a169]" -c:v libx264 -c:a aac output_169.mp4

PR engineering and gotchas

Testing the render contract

The tests I care about are not only "file exists." A bad ratio export can exist and still be unusable. The useful checks are visual and temporal:

• Safe-area assertions: captions and CTA blocks stay within platform-safe margins.
• Subject visibility: presenter face and product proof remain inside the frame at sampled timestamps.
• Audio parity: both outputs have identical voice duration and word-timing metadata.
• Frame count parity: 9:16 and 16:9 exports produce matching duration within one frame.

Those checks are cheap compared with manual review and catch the exact mistakes crop-after-render tends to hide until late QA.

What I'd do differently

I would unify captions around ASS/libass earlier instead of carrying canvas text layout per ratio. Canvas captions are workable, but subtitle tracks are naturally time-based and can adapt style per surface. The caption-rendering detail from Seam-Free Video Splicing belongs in the multi-ratio renderer as a first-class primitive.

I would also create a ratio matrix in the product spec before writing the renderer: 9:16, 16:9, 1:1, and maybe 4:5. Adding the second ratio forced the right abstraction. Designing for three or four would have made the layout contract clearer from day one.

The pattern: single audio, adaptive canvases

Multi-format AI video generation should preserve creative intent, not crop pixels. Keep one canonical audio timeline, render visual layouts per target surface, and mux each output from deterministic artifacts. That is how a 3-second hook can ship as both a vertical social ad and a widescreen placement without feeling like one was an afterthought.