A new benchmark has quantified something people building with AI video generation have suspected for a year: the models are excellent at producing footage that looks real and poor at producing the footage you asked for. SemComp-Bench scores video generation on two independent axes and finds that the best model reaches 91.8% on generation reliability while managing only 37.8% on outcome achievement. It was the number one paper of the day on Hugging Face, with 153 upvotes.
Key facts
- Best generation reliability score: 91.8%. Best outcome achievement score: 37.8%.
- 1,273 evaluation instances mined from the Koala-36M video dataset, with narration-dependent clips filtered out.
- Ranked #1 paper of the day on Hugging Face with 153 upvotes; the project page marks it as a research preview under review.
- Each clip is judged on 27 uniformly sampled frames.
Video model evaluation has been dominated by fidelity: does it look photographic, is the motion smooth, are there artifacts. Those are real qualities and they were the right thing to measure while models were still producing melting faces. They are also, unavoidably, what models have been optimized against for the last several generations. When you measure appearance and reward appearance, you get appearance.
SemComp-Bench splits the question in two. Each of its 1,273 instances gives a model a reference frame, a short instruction, and a longer detailed instruction, all drawn from a real source video whose actual outcome is known. The generated clip is then scored on two separate dimensions.
Generation Reliability is the familiar one, broken into five checks: is the result physically plausible, is it visually clear, is it free of rendering artifacts, is it spatially and temporally coherent within its scene, and does any text or interface content in the frame stay intact.
Outcome Achievement is the new one, broken into four: did the clip actually realize the instructed outcome, did it stay semantically grounded in the reference frame it was given, did the specific objects and people stay consistent throughout, and did overall visual continuity hold.
A detail in the protocol is worth calling out because it is the fair version of the test. The benchmark does not require the model to show the intermediate steps. If the instruction is to pour the coffee, a clip that cuts to a full cup is acceptable. What is not acceptable is arriving at the full cup by quietly replacing the mug with a different mug, jumping to a different kitchen, or drifting away from the reference image.
An analogy: hire someone to photograph your house being repainted. Ninety-two percent of the time you get a beautiful, sharp, professionally composed photograph of a house. Thirty-eight percent of the time it is your house, painted the colour you asked for.
The gap is the finding, and it is large enough that it cannot be explained away as evaluation noise. It says current video systems have learned the surface statistics of video extremely well and the causal structure of events much less so. Producing a plausible frame sequence and producing a sequence in which a specific thing happens to a specific object are different problems, and only one of them has been under sustained optimization pressure.
This lands alongside two other recent results pointing the same direction. A closed-loop benchmark caught nine world models forgetting the room they had just rendered, and separate work found frontier multimodal models cannot build a 3D world yet. Three independent benchmarks, three different methodologies, one shared conclusion: surface plausibility has outrun modelled consequence.
The honest caveat is that outcome achievement is judged by a model-based evaluation protocol, so a 37.8% ceiling is partly a statement about the judge as well as the generator. The authors mitigate this with a structured multi-check design rather than a single holistic score, and the reference-frame grounding gives the judge something concrete to compare against, but nobody should read 37.8% as a physical constant. The project page's own label -- research preview, under review -- is the appropriate amount of confidence.
The practical implication for anyone shipping generated video is direct. If your use case is atmosphere, texture, or b-roll, the fidelity numbers are the relevant ones and they are good. If your clip has to depict a specific result -- a product being assembled, a procedure being performed, a state changing -- benchmark on outcome, because your users will notice the difference long before your metrics do.
Related reading on this site: how AI is benchmarked, world models, LLM as a judge, and diffusion models.
Originally published on Ground Truth, where every claim is checked against the primary source.
