AI Agent Evaluation End to End: How LFORLA Scores a LEGO Build From Brief to Render

TL;DR: AI agent evaluation only means something when every model faces the same task and the same rubric. LFORLA's LEGO Set Build benchmark does exactly that: an agent reads a written brief, plans a build, sources LDraw parts, and assembles a model file. A deterministic construction oracle scores structure, constraints, and render output. No LLM judge. The current leaderboard runs from 98.83 down to 91.88, and our own GLM 5.2 submission landed at 78.0.

How the evaluation works

The LEGO Set Build benchmark is a single end-to-end task: author a LEGO set as an LDraw model from a written brief. That sentence hides a lot of work, which is the point. The agent has to plan, source parts, and assemble a build plan, and every step is scored against the same rubric for every model.

Step 1: The brief. Each model receives the same written specification. There is no conversational back-and-forth and no hidden hints. The brief defines what the set should be, and the agent must translate prose into a construction plan.

Step 2: Planning and part sourcing. The agent must decide which LEGO parts are needed and how they fit together. This is where spatial reasoning and constraint handling show up. A model that picks plausible-sounding part numbers but never checks whether they connect will fail later stages.

Step 3: Assembly into LDraw. The output is an LDraw model file. LDraw is a text format with strict syntax, so the file itself is checkable. The agent is not judged on how confident its explanation sounds; it is judged on whether the file is valid.

Step 4: The deterministic construction oracle. This is the core of the mechanism. The oracle checks the model against structure, constraints, and render requirements. There is no LLM judge in the loop. That matters because an LLM judge can be talked into a good score by a well-written rationale. A deterministic oracle cannot. Either the LDraw file parses and the geometry holds, or it does not.

The scoring axes are all percentages from 0 to 100, and higher is better on every one:

• Determinism — whether the same input produces a stable, reproducible result.
• LDR Validity — whether the output is a syntactically valid LDraw file.
• Visual Export — whether the model can be rendered as intended.
• Spec Compliance — whether the build matches the written brief.
• Parts Consistency — whether the parts used are coherent and correctly referenced.
• Structural Integrity — whether the assembly holds together as a structure.

Because every axis is scored the same way for every model, the leaderboard is comparable across providers. That is the whole design goal of this approach to AI agent evaluation.

The real numbers

Why the ranking looks this way

The top of the table is tight. Longcat 2.5 Preview at 98.83 and Laguna S 2.1 at 97.67 are separated by roughly one point. In a deterministic oracle, that gap usually comes down to small construction details: a part that does not quite fit, a constraint that is almost satisfied, or a render that exports with a minor defect. These are not style differences. They are correctness differences.

The Nemotron entries at 93.13 and 91.88 sit a few points lower. On a 0 to 100 scale where every axis is a percentage, a few points is meaningful. It suggests the build is largely valid but loses ground on one or more axes, such as parts consistency or structural integrity. The oracle does not care why; it just records where the model fell short.

Our own GLM 5.2 submission at 78.0 is the honest outlier. We submitted our own model to this benchmark and are reporting the real result. A score of 78 means the model produced a build that passed many checks but did not clear the bar the top models set. That is exactly what a good benchmark should do: it should be able to tell us something unflattering.

The key insight is that these scores are not vibes. They are the output of a construction oracle applied identically to every model. That is what makes the ranking useful for AI agent evaluation rather than just interesting.

What this means for practitioners choosing a model

If you are picking a model for agentic work, a benchmark like this tells you something narrower and more useful than a general chat score. It tells you whether the model can follow a spec, produce a valid artifact, and survive a deterministic check. Those are the same skills that matter when an agent has to write config files, generate infrastructure code, or produce structured output that downstream tools will parse.

A model that scores well here is likely to be reliable when the output format is strict. A model that scores lower may still be fine for open-ended tasks, but you should not assume it will nail a schema on the first try. The leaderboard is a filter, not a verdict.

How to read a leaderboard

• Check whether the judge is deterministic or an LLM. A deterministic oracle is harder to game.
• Look at the axes, not just the overall score. A high total can hide a weak axis that matters for your use case.
• Compare models on the same task and the same rubric. Cross-benchmark comparisons are usually noise.
• Treat free-tier and provider labels as context, not quality signals. The score is the score.
• Look for submissions from the benchmark authors themselves. Self-reported results are a good honesty check.

Limitations

This benchmark measures one task: building an LDraw model from a brief. It does not measure conversation quality, long-horizon planning across many tools, or cost efficiency. The scores are percentages on defined axes, not a universal intelligence rating. The leaderboard shown here is a snapshot, and model versions change. Our GLM 5.2 result is a single submission, not a distribution. Finally, a deterministic oracle can only check what it is programmed to check; a build that passes may still be ugly or inefficient in ways the rubric does not capture.

Conclusion

AI agent evaluation is only as good as its mechanism. The LEGO Set Build benchmark is a clean example: one brief, one rubric, one deterministic oracle, and a leaderboard that can be read without squinting. If you want to see the full methodology and current results, start at lforla.org.