Inside an Image-to-Video Motion Transfer Pipeline: From One Photo to a 9:16 Dance Clip

A developer's walkthrough of the stages behind "make this photo dance": pose extraction, conditioning, temporal consistency, and encoding for vertical video.

"Upload a photo, get a dancing video" is one of the easiest AI features to describe and one of the trickiest to make look good. Under the hood it's a pipeline of several separate problems, and each one can quietly ruin the output.

This article walks through that pipeline at an architectural level. It's based on the kind of system behind tools like AI Dance Generator, which turns one image into a 9:16 MP4. I'll stay model-agnostic, because the specific checkpoints change every few months while the structure stays the same.

The problem, stated precisely

Given:

• a source image I containing one subject, and
• a driving motion M, which is a sequence of poses p_1 … p_T,

produce a video V = (f_1 … f_T) where each frame shows the subject from I in pose p_t, with:

• identity preserved (face, body shape, clothing),
• temporal coherence (no flicker or texture crawl between frames), and
• plausible occlusion (parts of the body hidden in I must be invented convincingly when they come into view).

That last point is the hard part. A front-facing photo contains no information about the subject's back, so a turn requires the model to hallucinate it consistently.

Stage 1: Input validation and preprocessing

Don't spend GPU time on images that can't work. A cheap CPU or small-GPU pre-pass should:

1. Detect people and reject images with zero or several subjects (or ask the user to choose one).
2. Check coverage. Estimate keypoints and confirm the head and torso are visible. Missing limbs are recoverable; a missing head usually isn't.
3. Normalize. Fix EXIF orientation, convert to RGB, and resize to the model's working resolution while keeping the aspect ratio.
4. Segment the subject. A foreground mask lets you treat the subject and background differently later.

def preprocess(img):
    img = apply_exif_orientation(img).convert("RGB")
    people = detect_people(img)
    if len(people) != 1:
        raise InputError("Please upload a photo with exactly one person.")
    kp = estimate_keypoints(img, people[0])
    if not kp.has(["nose", "left_shoulder", "right_shoulder"]):
        raise InputError("Head and shoulders must be visible.")
    mask = segment_subject(img, people[0])
    return resize_keep_aspect(img, target=768), kp, mask

Returning a clear, specific error here is better UX than returning a broken video later.

Stage 2: Representing the driving motion

The dance is stored as pose data, not as video. Typical representations:

• 2D skeleton keypoints (OpenPose / COCO style): cheap and widely supported, but ambiguous in depth.
• Dense pose / UV maps: map each pixel to a body-surface coordinate, giving richer conditioning.
• 3D body model parameters (SMPL-like): handle rotation and depth best, but cost more to extract.

Motion is extracted once per dance from a reference clip, then cleaned up offline: smooth jitter with a one-euro or Savitzky–Golay filter, fill dropped keypoints, and trim to a clean loop.

Retargeting

The reference dancer and your subject have different proportions. If you feed raw keypoints, a short subject gets stretched limbs. Retarget by:

1. Computing bone lengths for the source subject from its keypoints.
2. Rescaling each bone in the driving sequence to match, while keeping joint angles.
3. Anchoring the root (mid-hip) to the subject's position and scale in the frame.

Skipping this step is one of the most common causes of "something looks off" results.

Stage 3: The generative core

Most current systems use a video diffusion model with two conditioning paths:

• Appearance conditioning: features from the source image, injected via a reference network, cross-attention, or both. This carries identity, clothing, and texture.
• Pose conditioning: the retargeted pose sequence, encoded by a lightweight pose guider and added to the noisy latents, similar in spirit to ControlNet.

Temporal layers (attention across frames) give the model a chance to keep frames consistent with one another. Frames are generated in latent space and decoded by a VAE.

Practical considerations:

• Clip length vs. memory. Attention across frames grows quickly with T. Long videos are usually generated in overlapping windows and blended at the seams.
• Sampling steps. Fewer steps mean faster jobs and more artifacts. Choose per product tier, not globally.
• Seed control. Fix the seed per job so retries are reproducible when you debug.

Stage 4: Temporal consistency and cleanup

Even good models produce flicker in high-frequency areas such as hair, patterned clothing, and hands. Mitigations, from cheapest to most expensive:

• Overlap-and-blend windows with a crossfade over the shared frames.
• Background locking: composite the generated subject over a stabilized version of the original background using the mask from Stage 1, so the background doesn't "breathe."
• Face restoration on a per-frame crop, blended back with a feathered mask. Use it lightly, because overdoing it creates a pasted-on look.
• Frame interpolation to raise the frame rate after generating at a lower one.

Stage 5: Framing for 9:16

The output targets vertical platforms, so composition matters:

• Track the subject's bounding box across all frames, then compute one smoothed crop path. Per-frame cropping causes a jittery virtual camera.
• Leave headroom. Platform UI covers the top and bottom of the screen.
• Pad rather than crop when the dance moves wide, so arms aren't cut off.

Stage 6: Encoding and delivery

Encode for maximum compatibility:

ffmpeg -framerate 24 -i frames/%05d.png \
  -vf "scale=1080:1920:flags=lanczos,format=yuv420p" \
  -c:v libx264 -profile:v high -crf 18 -preset slow \
  -movflags +faststart output.mp4

• yuv420p and H.264 High play everywhere.
• +faststart moves the metadata to the front so the video starts playing before it fully downloads.
• Add a silent audio track if a platform rejects video-only files, or mux in the dance's music where licensing allows.

System architecture

Generation takes seconds to minutes, so it must be asynchronous:

1. API receives the upload and runs Stage 1 synchronously, so bad input fails fast.
2. A job goes into a queue (Redis, SQS, or similar) with an idempotency key, so a double-click doesn't bill or render twice.
3. GPU workers pull jobs, run Stages 2–6, and upload the MP4 to object storage.
4. The client polls or subscribes to a status endpoint, then gets a signed URL.

This structure also fits per-output billing, which is what a pay-per-video product does: you charge when a job is created and finished, which lines up neatly with GPU cost per job.

Testing a generative pipeline

Unit tests won't tell you whether a dance looks good, but you can still automate a lot:

• Golden inputs: a fixed set of tricky photos (side profile, loose clothing, pet, low light) rendered on every model or config change.
• Cheap metrics: face-embedding similarity between source and frames (identity drift), and frame-to-frame difference in background regions (flicker).
• Human review of a contact sheet before shipping a new checkpoint.

Takeaways

• Most visible quality problems come from preprocessing, retargeting, and framing, not the diffusion model itself.
• Treat motion as clean, retargeted pose data prepared offline.
• Design for asynchronous, idempotent jobs from the start.
• Encode boring, compatible MP4s: 1080×1920, H.264, yuv420p, faststart.

If you want to see what this pipeline produces for a non-technical user, the make a photo dance flow is a compact example of hiding all six stages behind a single upload button.