GPQA Diamond Has 198 Questions: What the Hugging Face Leaderboard's 115 Rows Can and Cannot Rank

TL;DR

On October 22, 2026 the Hugging Face official GPQA Diamond leaderboard returned 115 rows for 97 repos, and 0 rows marked verified. 60 rows link to a model card, 22 come from an EvalEval source, 33 have another or no source.
GPQA Diamond has 198 questions. One question is 0.505 points. A score of 93.5% has a 95% sampling interval of roughly 89.2 to 96.2, so a model at 90.1% and a model at 93.5% are not separated by this test alone.
Of 93 rows whose values are rounded to two decimals or fewer, 57 cannot be a single pass over 198 questions (for example 93.5 is not a rounding of 185/198 = 93.43 or 186/198 = 93.94). Those rows are averages over runs or a different protocol. 43 of the 57 have an empty notes field, and only one says it is an average.
In the same Hugging Face repo, one model's score spans 67.1 to 80.1 across reasoning-effort settings on the model card. That is a bigger spread than the entire gap between rank 2 and rank 15.
In an illustrative simulation, a 2-point gap (92% vs 90%) survives a fresh draw of 198 questions about 77% of the time. It takes roughly 800 questions to reach 95%.

What is on the Hugging Face GPQA leaderboard?

The dataset repo Idavidrein/gpqa carries the tags benchmark:official and benchmark:eval-yaml, which mark it as an official benchmark that accepts eval YAML files. Models report results through .eval_results/*.yaml files in their own repos or through pull requests. The leaderboard shows what those files say.

I pulled the JSON the leaderboard view uses. I did not find documentation for that endpoint, so treat it as unstable. The snapshot (October 22, 2026, KST) looks like this:

Property	Count
Rows	115
Distinct repos	97
Rows with `verified: true`	0
Source name "Model Card"	60
Source name "EvalEval"	22
Other or no source	33
Rows with a free-text `notes` field filled in	63

Zero verified rows does not mean the numbers are wrong. It means the board is a collection of claims, and the only checks are the ones you do yourself. Everything below is about what you can check from the rows alone.

A roundup check first. An open-LLM roundup updated October 13, 2026 lists GPQA Diamond for eight models: 93.5 (Kimi K3), 91.2 (GLM-5.2), 90.5 (Kimi K2.6), 90.4 (Hunyuan Hy3), 90.1 (DeepSeek-V4-Pro), 88.1 (DeepSeek-V4-Flash), 87.8 (Qwen3.6-27B) and 87.0 (Nemotron 3 Ultra). All eight equal the model-card row on the Hugging Face board to the decimal. The roundup does not say where its numbers come from, but this match makes self-reported figures the likely source. I will use those eight as the running example, because they are the numbers people actually quote.

How much can 198 questions separate?

Treat each question as a coin flip with the model's true accuracy. The standard error of a score p over n = 198 questions is sqrt(p(1-p)/n). At p = 0.90 that is 2.1 points, and the 95% Wilson interval is about plus or minus 4 points.

Model (self-reported)	Score	95% interval over 198 questions
Kimi K3	93.5	89.2 to 96.2
GLM-5.2	91.2	86.4 to 94.4
Kimi K2.6	90.5	85.6 to 93.8
Hunyuan Hy3	90.4	85.5 to 93.8
DeepSeek-V4-Pro	90.1	85.1 to 93.5
DeepSeek-V4-Flash	88.1	82.8 to 91.9
Qwen3.6-27B	87.8	82.5 to 91.7
Nemotron 3 Ultra	87.0	81.6 to 91.0

Every interval overlaps with at least its neighbours. Overlap is a conservative test, and the better question is about the gap. Two models answer the same 198 questions, so their errors are correlated: a question that is hard for one is usually hard for the other. A correlation rho between the two models' per-question outcomes shrinks the noise in the gap to sqrt(2p(1-p)(1-rho)/n). The smallest gap that clears a 95% test is then:

True accuracy	rho = 0	rho = 0.5	rho = 0.8
80%	7.9	5.6	3.5
90%	5.9	4.2	2.6
93.5%	4.9	3.4	2.2

Nobody publishes the per-question correlation between two vendors' runs, so rho is an assumption. Even at an optimistic 0.8, a 90% model needs a 2.6 point lead before the order is statistically readable. On the board, 13 rows sit at or above 90.3, which is the lower end of the top row's own interval. The top of this board is a cluster, not a ladder.

Would the order survive a fresh draw of questions?

A simulation makes the point more concretely. I drew 198 questions with a shared difficulty factor (a latent Gaussian copula), set the accuracies to the reported values, and counted how often model A scored strictly higher than model B. These are illustrative numbers under the assumed correlation, not measurements of any real model pair.

Pair (reported scores)	Order holds, rho = 0.5	Order holds, rho = 0.8
93.5 vs 91.2	81.0%	85.9%
93.5 vs 90.1	90.8%	94.8%
91.2 vs 90.1	62.6%	66.0%
90.1 vs 88.1	74.4%	78.7%
93.5 vs 87.0	99.3%	99.9%

The 6.5 point gap between the first and last model is safe. Everything adjacent is closer to a coin flip with a thumb on the scale. Ties, which are up to 9% of draws here, count against "A is ahead" in this table.

The next chart holds the accuracy at 90%, sets rho to 0.6, and asks how many questions a benchmark needs.

For a 2-point gap the answer is about 800 questions (94.6%) and for a 1-point gap more than 1,600 (87.8%). GPQA Diamond was built as a hard, expert-validated subset, so its small size is a design choice, not a mistake. The consequence is that it was never built to rank models 2 points apart.

Are these rows single runs over 198 questions?

This is the check I find most useful, and it takes four lines of code. If a row is one pass over the 198 questions, its value must be k/198 for an integer k. The rows from EvalEval are: all 22 have long decimals such as 87.8787878788, which is exactly 174/198. Their values come straight from a count.

The other rows are rounded by their authors. For a value rounded to two decimals, the chance of landing on a k/198 value by coincidence is about 2%, so a hit is meaningful. For one decimal it is about 20%, so a hit proves little, but a miss is conclusive:

93 rows have two decimals or fewer.
36 of them are consistent with one pass over 198 questions.
57 are not.

The headline example is 93.5. The nearest single-pass values are 185/198 = 93.43 and 186/198 = 93.94. A reported 93.5 therefore comes from an average over several runs, a different item count, or a different aggregation. The row's notes field is empty, so the board does not say which. Across all 57 off-grid rows, 43 have no notes and only one ("Avg@4") names an averaging protocol.

Why it matters: an average over several samples has less noise than a single run, but a majority vote over up to 16 samples or a larger thinking budget is a different measurement. Notes on the board range from "greedy decoding, single sample" to "majority vote over up to 16 samples" to "Avg@4". They all land in one column and get ranked together. I cannot tell you which protocol is best. I can tell you the column does not hold one quantity.

Same repo, different setting: how big can the gap be?

Fourteen rows belong to the two gpt-oss repos: each model card reports three reasoning-effort levels, each with and without tools (six rows), plus one EvalEval row per repo. Without tools, the model-card values are:

Reasoning effort	gpt-oss-120b	gpt-oss-20b
low	67.1	56.8
medium	73.1	66.0
high	80.1	71.5

Same weights, spread of 13.0 points for the larger model and 14.7 for the smaller. For comparison, the whole distance from rank 2 (93.5) to rank 15 (89.5) on the board is 4.0 points. A setting you did not read can move a score more than every model gap in the top 15.

The EvalEval rows show the same trap from the other side. For gpt-oss-120b, EvalEval reports 80.81, which sits next to the "high" model-card value (80.1). For gpt-oss-20b, EvalEval reports 58.59, which sits next to the "low" values (58.0 and 56.8) and 12.9 points below the "high" card value. The EvalEval row records no reasoning effort, so a naive comparison would conclude that the 20b card is inflated by 13 points. More likely the two sources ran different settings. I cannot confirm that from the rows, and that is the point.

Do self-reports run higher than independent re-measurements?

Four repos have both a model-card row and an EvalEval row, with a setting I can match:

Repo	Model card	EvalEval (single pass)	Difference
Kimi-K2.5	87.60	87.37	+0.23
Qwen3.5-122B-A10B	86.60	84.85	+1.75
Qwen3.5-27B	85.50	81.82	+3.68
gpt-oss-120b (high, no tools)	80.10	80.81	-0.71

The mean difference is +1.24 points, positive in three of four. A single pass over 198 questions at 85% accuracy has a standard error of 2.5 points, so only the Qwen3.5-27B gap (1.5 standard errors) is anywhere near notable, and 4 repos cannot tell a 1 point inflation from none. I read this as "no evidence of large inflation, and no power to see a small one". It is not a verdict on any vendor.

Do repeated numbers mean copied numbers?

Seventeen values appear under more than one repo. Most are coincidences at a 0.1 resolution among 115 rows, or sibling checkpoints. Three look different:

zai-org/GLM-5.1 and two MXFP4 AutoRound re-uploads (INCModel3/... and intel-ai/...) all list exactly 86.2.
nvidia/...Nemotron-3-Super-120B-A12B-BF16 and RedHatAI/...Nemotron-3-Super-120B-A12B-BF16 both list the same two values, 82.7 and 79.23.

A quantized checkpoint reproducing its parent's score to the decimal is more plausibly a copied figure than a re-measurement. That is an inference, since the rows do not say. If true, those rows inflate the count of "models above 86" without adding evidence.

Is there headroom left to rank at the top?

GPQA Diamond is a ceiling-limited test. The original paper reports expert accuracy of 65% on the full 448-question set, 74% when excluding errors the experts later identified. An Epoch AI analysis from May 30, 2025 hand-reviewed six of the hardest Diamond questions, judged about 2.25 of them invalid, and extrapolated to roughly 8% of the benchmark (about 15 questions). That extrapolation rests on six questions and carries a wide error bar.

Take it at face value anyway. Fifteen invalid questions would put the ceiling at 183/198 = 92.4%, and 5 of the 115 rows are above it, including the top row at 94.44 (187/198). A model can match the key on a flawed question, for example by reproducing the same common mistake as the answer key, so being above the "valid ceiling" is not proof of anything wrong. It does mean that at the very top, a point or two measures key agreement as much as science knowledge.

A six-step check before quoting a GPQA number

Read the notes field. If it is empty, you do not know the protocol.
Test the grid. If the value is not k/198 (and it has at most two decimals), it is not one pass.
Find the setting: reasoning effort, tools, temperature, number of samples. Compare only matching settings.
Attach an interval. For n = 198 use plus or minus 4 points near 90%.
Check whether the repo is a quantized re-upload of a model already on the board.
Say "tied within noise" when the gap is under about 3 points, unless you have per-question data.

The script below reproduces the board statistics, the grid test and the interval table. It needs numpy only for the simulation, which I left out here for brevity.

import json, math, urllib.request, collections
URL = "https://huggingface.co/api/datasets/Idavidrein/gpqa/leaderboard"
rows = json.load(urllib.request.urlopen(
    urllib.request.Request(URL, headers={"User-Agent": "Mozilla/5.0"}), timeout=60))
N = 198

def decimals(v):
    s = repr(float(v)); return len(s.split(".")[1].rstrip("0")) if "." in s else 0

def on_grid(v):
    d = min(decimals(v), 2)
    return any(round(100 * k / N, d) == round(v, d) for k in range(N + 1))

def wilson(p, n=N, z=1.96):
    c = (p + z*z/(2*n)) / (1 + z*z/n)
    h = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / (1 + z*z/n)
    return round(100*(c-h), 1), round(100*(c+h), 1)

print(len(rows), len({r["modelId"] for r in rows}), sum(r["verified"] for r in rows))
low = [r for r in rows if decimals(r["value"]) <= 2]
print("rounded rows:", len(low), "off the k/198 grid:", sum(not on_grid(r["value"]) for r in low))
print("93.5 ->", wilson(0.935))

Limitations

One snapshot, taken October 22, 2026. The board changes as pull requests merge.
The grid test cannot say why a value is off the grid. Averages, a different item count and partial-credit scoring all produce it.
The simulation assumes a latent Gaussian copula with rho set by hand. Real models are not Gaussian, and real rho is unknown.
The card-versus-EvalEval comparison has four repos and a single EvalEval pass each.
I ran no model. Every score is as published.

FAQ

How many questions are in GPQA Diamond?

It is the hardest subset of the 448-question GPQA set, kept to questions where both expert validators answered correctly. One question is 0.505 points of accuracy.

Is a 93.5% GPQA Diamond score better than 90.1%?
Probably, but not by this test alone. Over 198 questions the 95% interval for 93.5% runs from about 89.2 to 96.2, which contains 90.1. In the simulation above, the order holds in 91% to 95% of fresh draws depending on the assumed correlation.

What does "verified: false" on the Hugging Face board mean?
Every one of the 115 rows had it, so the board gave no independent confirmation for any score. The rows record a claim and its source link, not a reproduction.

Why would a GPQA score not be a multiple of 1/198?
A single pass over the 198 questions produces only multiples of 1/198 (0.505%). A value off that grid means averaging over several samples, another item count, or a different aggregation.

How should I compare two open models on GPQA Diamond?
Match the setting first: reasoning effort, tools, temperature, number of samples. Then compare with an interval. If the gap is under about 3 points, call it a tie unless you have per-question results for both models.