We were about to ship a performance write-up claiming three queries had gotten 2-4x slower. All three numbers were wrong, and the reason is one every benchmark author should worry about: the same binary, on the same machine, running the same query, gave different answers depending on what time of day we ran it.
The measurement that didn't add up
While checking a change to Samyama Graph, we timed LDBC SNB Interactive query IC9 on one commit, one dataset, one 16-core workstation, four hours apart:
| when | commit | IC9 median |
|---|---|---|
| morning | b2a3557 |
2,822 ms |
| evening | b2a3557 |
4,912 ms |
Same binary. 1.74x slower, four hours later. ps showed the benchmark pinned at 100% of one core and nothing else on the box above 9%. No other process explains that gap.
So we ran it again — same binary, minutes apart this time, nothing changed in between:
| run | IC9 median |
|---|---|
| 1 | 2,011 ms |
| 2 | 2,490 ms |
24% apart. Back to back.
Why we almost missed it
We had a real change on main to compare against a baseline on an older commit. Baseline was measured in the morning; the new code was measured that evening. Read naively, the evening numbers said IC10 had gotten 2x slower, IC11 4x, IC12 3x.
None of those regressions existed. The host itself had drifted between the two sittings — thermal throttling, frequency scaling, background load, whatever the cause, the machine at 6pm was not the machine at 9am. We re-ran everything back to back, on one sitting, and the picture reversed:
| query | before (b2a3557) |
after (main) |
ratio |
|---|---|---|---|
| IC9 | 4,912 ms | 2,490 ms | 1.97x faster |
| IC10 | 340 ms | 352 ms | ~1.0x |
| IC11 | 382 ms | 400 ms | ~1.0x |
| IC12 | 428 ms | 483 ms | ~1.1x — inside the 24% band |
The change was a genuine ~2x win on IC9 and a wash everywhere else. The "2-4x regressions" were the host, not the code.
We only caught it because a 4x regression from a small change looked implausible enough to re-measure. A smaller, real regression sitting inside that same noise band would not have raised anyone's eyebrows, and would have shipped.
What this means for anyone reading benchmark numbers
Two things, and they cut in opposite directions:
A regression can hide inside the noise. If your benchmark's run-to-run variance is 24%, any real regression under ~25% is statistically invisible next to it. A nightly CI gate set to fire on a 25% threshold would page on noise some nights and miss a real 20% regression every other night.
An improvement can be manufactured by timing alone. Measure a baseline in the morning and a new version in the evening on a drifting host, and you can produce whatever ratio you want without touching a line of code. This is true whether the drift is accidental or not — which is exactly why "before" and "after" have to come from one sitting.
Neither of these is specific to our engine, our benchmark suite, or Rust. It's a property of measuring wall-clock time on a shared physical machine, and it will bite any project that compares two numbers taken hours apart and calls the difference a result.
What we changed
A calibration line in every run. Each benchmark now times a short, fixed, CPU-bound operation at the start and the end, and reports it alongside load average and mean core frequency. Two runs whose calibration numbers disagree were taken on hosts running at different speeds, whatever the milliseconds of the actual queries say. If a single run's own opening and closing calibration differ by more than 10%, the run says so — its numbers aren't even internally comparable to each other.
A rule for before/after claims. A performance comparison needs both numbers from one back-to-back session on one machine — not "the same machine," the same sitting. Quote the ratio, not the absolute milliseconds, when the point is an improvement: a ratio measured back to back survives a slow host; an absolute number doesn't survive being read next to one taken elsewhere.
Cross-engine ratios get the same treatment. If a "we're 3x faster than X" claim has our number from one session and the competitor's from another, the ratio isn't a ratio — it's two unrelated numbers divided by each other. Same machine, same sitting, or the comparison doesn't get published.
The full account, including the four numbers that started this, is in the original issue: samyama-ai/samyama-graph#529. The benchmark suite and the LDBC SNB reproducer that surfaced this are in the repository — cargo bench --bench ldbc_benchmark, documented in docs/BENCHMARKS.md.
We'd rather publish this than the regression that wasn't real.
