Most of us met generative video through text-to-video: you write a paragraph, cross your fingers, and get back something that is almost what you meant. If you come from software, that workflow feels oddly unstructured. There's no input contract, no deterministic part, and every retry is a full re-run.
Video-to-video models flip this around, and once I started thinking about them like a function call instead of a wish, my results got noticeably better. Here's the mental model.
The signature
Think of a reference-driven video model as something like this:
generate(
source: Video, // motion, timing, camera
refs: Image[], // who / what appears
prompt?: string, // the glue: which ref goes where
mode: "motion" | "swap",
resolution: "480p" | "720p" | "1080p"
): Video
The key insight: the source video is not a suggestion, it is the bulk of your specification. It already encodes choreography, pacing, framing and camera movement, which is exactly the stuff that is painful to describe in words. Your prompt shrinks from a screenplay to a few lines of wiring.
Two modes, two very different contracts
Motion transfer keeps only the movement and camera of the source and rebuilds everything else from your images. Contract: "same motion, new world."
Object swap keeps the original footage and replaces only the element you point at: a bottle, a jacket, a car, a person. Contract: "same world, one diff."
If you think in diffs, object swap is the git commit that touches one file. Motion transfer is a rewrite that keeps the interface.
Positional references: the @Image1 pattern
Some tools number your uploaded references and let you cite them in the prompt, e.g. @Image1, @Image2, plus @Video for the source. This is basically positional arguments, and the same habits that make function calls readable apply:
@Image1 dances like the person in @Video, wearing the jacket from @Image2
replace the bottle with @Image3
A few rules that held up in my testing:
- Keep the prompt about mapping, not motion. Don't redescribe the dance. The source already "knows" it. Re-describing motion is like passing a value that's already in scope; at best it's noise, at worst it conflicts.
- One responsibility per reference. A face shot for identity, a separate flat-lay for the outfit. Mixing roles in one image makes it ambiguous which attribute the model should extract.
- Order matters. If a template ships with sample references, your uploads typically replace them in order. Treat it like a function signature: argument 1 stays argument 1.
Validate your inputs
Garbage in, garbage out applies more strongly here than with text prompts, because the source video carries so much of the spec.
- Source clip: one continuous shot beats a fast montage. Hard cuts are like unhandled exceptions: the model has to guess what happens across the boundary. For swaps, the target object should stay visible for most of the clip.
- Reference images: well lit, subject filling a good part of the frame, front-facing for faces, plain background for products. Add a second angle if the subject turns away from camera.
- Length: tools usually bound this (4–30 seconds is a common window). Shorter clips also mean cheaper iterations.
Iterate like you would with a slow CI pipeline
Generation is slow and costs money, so treat it like an expensive build:
- Draft at low resolution. Run 480p to check identity, framing and that the right reference landed in the right place. It's your unit test.
- Promote the winner. Only re-render the keeper at 720p or 1080p.
- Change one variable at a time. Swap the reference or the prompt or the source, never all three, or you can't tell what fixed (or broke) the output.
- Log your runs. I keep a tiny JSON file per experiment: source clip, reference filenames, prompt, mode, result notes. It sounds overkill until you try to reproduce a good result a week later.
{
"source": "runway-walk-01.mp4",
"refs": ["face_front.jpg", "coat_flat.jpg"],
"prompt": "@Image1 walks like the person in @Video, wearing @Image2",
"mode": "motion",
"res": "480p",
"notes": "identity stable, coat collar drifts at 3s; try second coat angle"
}
Trying it out
If you want to poke at this without wiring up any infrastructure, I've been experimenting with Genjutsu, a browser-based video-to-video generator that exposes both modes and the numbered-reference prompting described above, and shows the credit cost before each run. Whatever tool you use, the mental model transfers.
Responsible inputs
One non-technical note: only use faces, products and footage you have the rights to. "It's just a test" stops being true the moment a clip gets shared.
Takeaway
Stop treating the prompt as the whole program. In video-to-video, the source clip is the spec, the references are the arguments, and the prompt is the wiring. Write it that way and your retries go down.
What patterns have you found for making generative video less of a slot machine? Drop them in the comments.
