How LLM Evaluation Actually Works: Inside the Satellite Geo QCM Leaderboard

TL;DR: LLM evaluation only produces comparable numbers when every model faces the same prompts, the same scoring rules, and a public trail you can re-read. The Satellite Geo QCM benchmark does exactly that: 4-choice geolocation from satellite images, deterministic scoring, no LLM judge. On the public leaderboard, DeepSeek V4 Flash Vision (exp) scores 98.18, while our own submitted model, GLM 5.2, scores 78.0. Here is the mechanism behind those numbers.

How the evaluation works

LLM evaluation is often treated as a black box. This benchmark is not. The mechanism is simple enough to reproduce by hand.

1. Same prompts for every model. Each item is an aerial or satellite image paired with a single question: pick the correct location among 4 options. The options are fixed. The wording is fixed. No model gets a hint, a retry, or a different framing.

2. A fixed judge — in this case, no judge at all. The benchmark is fully deterministic. There is no LLM judge scoring free-form text. The model outputs one of four choices, and the answer is either right or wrong. That removes a huge source of variance: judge drift, prompt sensitivity, and model-specific grading quirks.

3. Per-axis rubrics. Scoring is broken down by axis. Overall accuracy is reported, and accuracy is also reported by difficulty. That means a model that handles easy items but collapses on hard ones shows up differently from a model that is uniformly mediocre.

4. A public verbatim trail. Anyone can re-read what a model actually answered. This is the part most leaderboards skip. Without the verbatim trail, you are trusting a number. With it, you can inspect the exact image, the exact options, and the exact choice the model made.

The real numbers

Why the ranking looks this way

The gap between 98.18 and 78.0 is not a rounding artifact. It is roughly 20 percentage points of accuracy on a deterministic task. In practice, that means DeepSeek V4 Flash Vision (exp) is picking the right location almost every time, while GLM 5.2 is missing about one in five items.

Because the benchmark has no LLM judge, the score is not an opinion. It is a count of correct choices divided by total items. The 98.18 figure is a raw accuracy percentage. The 78.0 figure is the same kind of number for our own model.

The per-difficulty axis matters here. A model can score well overall by dominating easy items and still fail on hard ones. If you only look at the headline number, you miss that shape. The public verbatim trail is what lets you check whether the misses are clustered on a specific kind of image or location.

What this means for practitioners choosing a model

If you are selecting a vision model for geolocation, a leaderboard like this is a starting point, not a verdict. The deterministic scoring means the number is trustworthy as a measure of this task. It does not mean the model will win on your task.

Use the per-axis breakdown to match the benchmark to your workload. If your images are mostly easy, a high overall score may be enough. If your images are hard, look at the difficulty axis before you commit.

Use the verbatim trail to sanity-check failures. A model that misses because it misread the options is a different problem from a model that misses because it cannot localize terrain.

How to read a leaderboard

• Check whether the judge is deterministic or an LLM. Deterministic scoring is easier to compare across models.
• Look for per-axis or per-difficulty breakdowns, not just a single overall score.
• Find the verbatim trail. If you cannot re-read what the model answered, treat the number as a claim, not evidence.
• Confirm that prompts and options are identical across models. Any per-model prompt tuning breaks comparability.
• Compare models on the axis closest to your real workload, not on the headline ranking alone.

Honest limitations

This benchmark measures one narrow skill: picking a location from a satellite image among 4 options. It does not measure reasoning, instruction following, or open-ended vision. The deterministic format is a strength for comparability and a limit for realism. A 4-choice format can also be gamed by guessing, so the difficulty axis and the verbatim trail are essential context. Finally, a single leaderboard entry is a snapshot, not a permanent ranking.

Conclusion

LLM evaluation becomes useful when the mechanism is visible: same prompts, fixed scoring, per-axis rubrics, and a public trail. The Satellite Geo QCM leaderboard shows what that looks like in practice, with DeepSeek V4 Flash Vision (exp) at 98.18 and our own GLM 5.2 at 78.0. You can inspect the full leaderboard and the benchmark details at lforla.org. If you are running your own evaluation, start by making your mechanism reproducible — then publish the trail.