I've been building an autonomous art director. Give it a one-line brief, a planner turns that into a shot description, SDXL with Juggernaut XL v9 renders it, a vision model looks at the output and writes fix directives, and the loop re-rolls until the critic clears the bar. Then a video model animates the winning frame.
Hardware is one RTX 3090, 24GB, a 5800X and 64GB of DDR4. That card is the constraint everything else bends around.
The first loop looked perfect
The loop ran end to end at 1:30am on 10/16. Planner, render, critique, verdict, 38 seconds for a full round. The first brief I gave it, a neon rain heist street, cleared at round one with a 9/10. Artifacts on disk: final frame, timeline HTML, run JSON, every round saved.
That should have been suspicious. A critic that hands out 9/10 on the first pass is a rubber stamp with a vision encoder. The loop had a bar of 8 and an or verdict == pass clause in the clear check, so a generous model could wave anything through.
The video model raised the bar by failing
Then I moved to the video half, MiniMax-H3 image-to-video on the same card. First attempts came back as mud. Then the recipe cracked and scores went 2/10, 4/10, 9/10 across three takes. The fix that mattered was CLIPLoader type minimax instead of hidream, plus the official sampler chain: RandomNoise and BasicGuider with no negative prompt and no CFG, KSamplerSelect res_multistep, BasicScheduler simple at 8 steps, the turbo 8-step LoRA through LoraLoaderModelOnly, and the latent pulled from slot 1 of the MiniMaxH3ImageToVideo node. Take 3 held geometry and showed no banding.
So I had a 9/10 clip. I put it in front of the person who has to approve the demo and got "fake as fuck, minecraft rain physics" back. Fair. Heavy explicit rain streaks render as particle sprites, and real night rain reads as mist, beads, drips, puddle rings.
That verdict forced a second look at all of it, and the second look was worse.
Frame stepping is a different measurement
I ran an independent QC pass that ignored the blind score. 13 spread frames per clip plus two 7-frame consecutive runs, stuck onto contact sheets, read frame by frame. Take 6, the max-tier 50-step render, scored 3 to 6 out of 10 in most cells with motion at 4/10. Pseudo-glyph signage morphed between frames. The umbrella figure got re-synthesized every frame like the model was redrawing it from scratch. Ground speckle boiled. Rain read as absent or as sprites. Take 3, my 9/10, scored 4 to 6, with a vendor face hallucinating mid-clip and steam teleporting between frames.
At playback resolution the blind score had been accurate: geometry locked, no banding. It never looked at two consecutive frames and asked whether they belonged to the same world.
Three root causes, in the order they matter. The source still broke the recipe I'd pinned myself, with a human figure and legible signage in frame, and the video model reinvents both every frame. The hi-res pass was soft: lanczos upscale plus img2img at 0.42 denoise left a laplacian of 29.7 against the clip's frame 0 at 30.9, so the model animated mush and preserved its mushiness frame by frame. Generic image-to-video temporal noise did the rest.
The memory wall behind it
Every attempt to fix this by brute force hit the card. 1920x1088 OOM'd because the CFG path needs two forward passes per step, so the hires tier doubles memory before it improves anything. 1664x960 fits at 21.6GB of 24GB against a 1.2GB baseline, and only with the card actually clear. More than once it wasn't: llama-swap and the resident engine crept back onto the GPU mid-render, and a 10:30 cron following stale worklog state is my main suspect.
The nastiest failure was the 124-frame VAE decode at 1664x960 spilling into host RAM. The box went into a swap death spiral where ICMP answered, TCP ports answered, ssh died at banner exchange, and ComfyUI returned an empty HTTP body. A machine that answers a ping and can't say hello has run out of memory, and the network was fine the entire time.
The guards
Four changes landed in the loop, committed as c26c82c.
Bar 8 to 9 with a strict clear: score at least 9 AND a pass verdict. The or verdict == pass bypass is gone.
Chain-aware taste in the planner and the critic. No people or creatures, no legible text or glyphs, ambient motion potential, clean at 1080p. A violation caps the score at 7 no matter how pretty the frame looks.
Fix deltas reach the planner. They were being computed and then discarded, which made the loop decorative. Now the planner receives the problems, the prompt and negative deltas, and the recent prompts, and has to evolve them.
The hires pass became a latent upscale, 1.5x lanczos plus a 30-step KSampler at 0.42 denoise, dpmpp_2m with karras, cfg 5, replacing the soft lanczos plus img2img route.
The gate standard changed too: no clip ships on a single-frame read at playback resolution. 13 spread frames plus consecutive runs, every time.
Audio got its own version of the same treatment. The generated rain audio passed an independent realness score of 0.91, "real or pro-produced", which was encouraging, but the master measured -30.89 LUFS with a -17.33 true peak. Two-pass loudnorm to I=-14, TP=-1.5 landed at -14.7 LUFS, -1.8 dBTP, zero clipped samples. That recipe is a pipeline constant now.
What shipped and what's next
The hardened loop is in. No clip passes the video gate yet. The winner has to come from a still that obeys the chain recipe from the start, rendered by the new loop at bar 9, upscaled properly, then animated at 50 steps with CFG 2.0 and no LoRA, which runs about 65 minutes per clip.
The second 3090 is seated in the lower slot. One card carrying the render and the critic at the same time is why the resident engine kept getting evicted mid-run, and why every hires tier turned into a memory negotiation. Two cards put the critic on its own 24GB and end the competition for VRAM. The machine is down for a power supply swap, and when it comes back the first job is the same brief at the higher bar.
A day and a half of measuring taught me one cheap habit. A model grading its own output at playback resolution will call the frame beautiful. Ask it about frame 47 and frame 48, and it tells you whether the frame holds still.
