I Let an LLM Art-Direct Lyric Videos. Then I Took Most of the Job Away.

I've been building LyricVideoGenerator, a tool that takes a song and its lyrics and renders a lyric video where each word lights up as it's sung. Most of the people using it make songs with Suno or Udio and want something to post on YouTube that isn't a static cover image.

The pitch is that you don't design anything. You drop in a song and get three finished looks to pick from. Somebody has to make those three looks. My first answer was "the LLM does it," and it didn't survive testing. This post is about what replaced it.

TL;DR: Letting the LLM design the whole look worked, but took up to a minute and kept producing the same three designs. Now the LLM only labels the song (mood, genre, energy, chorus lines), and plain code picks three looks from a library of hand-built presets.

The renderer only reads JSON

The video is a Remotion composition, and the only thing it reads is a Style object:

type Style = {
  template: 'focus' | 'scroll' | 'caption';
  palette: { bg: string; fg: string; accent: string; muted: string };
  typography: { family: string; weight: 400 | 500 | 600 | 700 | 800 | 900;
                scale: number; transform: 'none' | 'uppercase';
                align: 'left' | 'center' };
  background:
    | { kind: 'solid' }
    | { kind: 'gradient'; from: string; to: string; angle: number }
    | { kind: 'cover-blur'; blur: number; dim: number }
    | { kind: 'image'; url: string; dim: number };
  animation: 'word-highlight' | 'karaoke' | 'line-fade'
           | 'word-pop' | 'typewriter';
  sections: { chorus: SectionStyle; verse?: SectionStyle; /* ... */ };
  // plus emphasis words, a cover badge flag, a name and a one-line rationale
};

So "design a lyric video" really means "fill in this object." That made it look like a perfect LLM job: a closed schema, a bounded set of choices, and taste as the only hard part.

Version 1: the model designs everything

The first prompt described what each field does on screen (the focus template shows one line at 110px, caption puts a subtitle block in the lower-left third, and so on) and asked for three complete Style objects per song.

Zod validation alone wasn't enough. Only about 4% of calls failed on structure, but the output was still often unusable. The failures that mattered were about meaning.

• Fonts that don't exist. Big Shoulders Display showed up twice; Google Fonts merged it into Big Shoulders a while ago. I ended up shipping a list of all 1,950 Google Fonts families and rejecting anything not on it.
• Highlight you can't see. For cover-based looks it would pull two near-white colours from the album art and use one for the lyrics and one for the highlighted word. On screen you couldn't tell which word was being sung.
• Half an empty screen. It paired the caption template with a flat colour background. caption exists to leave the top two-thirds of the frame to the cover art, so over a solid colour you get half a screen of nothing. That was 3 of 18 candidates in one round.

Each of those turned into a check: a font allowlist, contrast thresholds (lyrics at least 4.5:1 against whatever is actually behind them, the highlight colour at least 3:1), and "caption only over cover backgrounds." For background images I compute the real colour behind the text with sharp (average and brightest colour of the cover, blended with the overlay at its dim value) instead of trusting the model's bg field.

The rule I held to: code validates and repairs, it never makes visual decisions.

With that in place it worked. Across 6 songs × 3 runs, every run produced 3 valid candidates within one retry, and when I went through the 18 chorus frames, 15 were usable as-is.

Why it still wasn't good enough

Two problems that more validation couldn't fix.

Latency. Three full designs took 40–60 seconds at the median and up to ~100 seconds. The cost was fine (two or three cents per song). Waiting a minute after uploading a song is not.

Sameness. Every song got its own three looks, but across songs the pattern was obvious. Over 54 candidates the template was locked to the slot: the first was always focus, the third always caption. The "bold" option was nearly always a dark gradient, centred heavy type and an amber highlight. Anyone making their third video would notice the AI has three moulds.

I tried prompt work on the second problem. Removing hard-coded suggestions and asking for genre-specific choices raised font variety from 12 families over 18 candidates to 15 or 16, but the template-by-slot pattern never went away.

Version 2: the model tags, code picks

What runs in production now splits the job:

1. One cheap LLM call reads the lyrics and style tags and returns only labels: 1–3 moods, 1–2 genres, an energy level, which lines are the chorus, and 3–6 emphasis words.
2. Code scores every hand-built preset against those labels.
3. Code picks the best set of three under some diversity rules.
4. Presets marked "from cover" pull their palette from the album art, and everything goes through the same validator as before.

The scoring is deliberately boring. Mood counts most, then genre, then how close the energy level is:

// weighted overlap: the first mood/genre the LLM lists counts most
const hit = (tags: string[], picked: string[], w: number[]) =>
  picked.reduce((s, x, i) => s + (tags.includes(x) ? w[i] : 0), 0) /
  picked.reduce((s, _, i) => s + w[i], 0);

export function scorePreset(p: Preset, a: Analysis) {
  const m = hit(p.tags.mood, a.mood, [1, 0.7, 0.5]);
  const g = hit(p.tags.genre, a.genre, [1, 0.6]);
  const d = Math.abs(ENERGY_ORD[p.tags.energy] - ENERGY_ORD[a.energy]);
  const e = d === 0 ? 1 : d === 1 ? 0.4 : 0;
  return 0.45 * m + 0.35 * g + 0.2 * e;
}

Picking three is a brute-force loop over every triple. With 59 presets that's about 32,000 triples, which is nothing. A triple is only allowed if all three templates differ or all three animations differ, and if the song has cover art at least one pick has to use it. Triples that differ on both get a small bonus.

What I gave up: the model no longer invents a look nobody has seen before.

What I got: the slow part of the call is gone, every option has been checked by a human eye at least once, and variety now depends on the preset library. Growing that library is my job and doesn't depend on prompt luck.

If the tagging call fails twice, a rules-only fallback picks from the style tags and the song's line density, so the user always gets three options.

The LLM turned out to be good at the labelling. With section markers stripped out, it called "is this line part of the chorus" correctly on all 178 lines in my test set, twice. Full section labels were 97–98% right. The misses were bridge vs. pre-chorus confusions, which don't change anything visually, because the styling only cares whether a line is in the chorus.

Where the LLM still edits the design

There's one place the model still touches the Style directly: "make the chorus bigger," "warmer colours," "less busy." That request goes to the model along with the current Style, and the model returns an RFC 6902 JSON Patch instead of a new object:

{
  "patch": [
    { "op": "add", "path": "/sections/chorus/typography/scale", "value": 1.35 }
  ],
  "explanation": "Made the chorus lines larger so it lifts."
}

A patch keeps the edit minimal, and the change is easy to show and revert. Every patch is stored as a revision on the project. One gotcha: an add fails if the parent object doesn't exist, and "add /sections/chorus/typography/scale to a chorus with no typography yet" is exactly what the model is told to emit. So before applying the patch I create the missing parent objects. The patched result goes through the same validator, so a "make it moodier" request can't produce unreadable text.

Timing, briefly

The word-by-word timing doesn't use an LLM at all. The audio and the lyric lines go to a forced-alignment API, which returns start and end times plus a per-word loss value. I map those timings back onto our lines by character offset and turn the loss into a 0–1 confidence with a logistic curve (calibrated so a loss of 1.8 maps to 0.6). Lines below 0.6 get flagged in the editor so you can nudge them by hand. Forced alignment is good, but a held note or a backing vocal can still throw it off.

What I'd tell myself at the start

• If an LLM's output is going to be rendered, write the semantic checks before the prompt. Valid JSON can still be an unreadable frame.
• "The model designs it" demos better than "the model picks from things a person designed." In daily use the second one is faster and repeats itself less.

If you want to see the result, the free tier of LyricVideoGenerator exports 720p 16:9 without an account.