You can merge a patch. You cannot merge trust.
An AI suggested a new parser. It promised forty percent fewer allocations. My profile agreed. The first run looked great. Then I plotted it. The graph told the truth. The speedup was a warm cache, not a better algorithm.
I built this experiment while testing MonkeyCode's free model access and its free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The tool started the conversation. The chart ended it.
Why one number is a lie
Benchmarks are graphs, not decimals. A single timing is a sample of noise. It includes warm-up, frequency scaling, background jobs, and the phase of the moon. Run the same binary twice and you get two answers. Run it twenty times and you get a distribution.
So why did the parser look forty percent faster? Because I measured one input size. I measured one run. I measured the size the AI happened to test first.
Change the input size and the story changes. Bigger inputs move the bottleneck. Allocation patterns shift. Caches overflow. The curve bends.
That is the real question: not "is it faster?" but "where is it faster?"
The workflow I now use
For every AI-proposed optimization, I run a size sweep. I take the old binary and the new binary. I feed them identical inputs across five growing sizes. I run each size five times and take the median.
Here is the script I keep next to every such patch:
#!/usr/bin/env python3
import subprocess
import statistics
import time
def bench(binary, data, runs=5):
samples = []
for _ in range(runs):
t0 = time.perf_counter()
subprocess.run([binary], input=data, capture_output=True)
samples.append(time.perf_counter() - t0)
return statistics.median(samples)
sizes = [10_000, 30_000, 100_000, 300_000, 1_000_000]
for n in sizes:
# reproducible synthetic input
data = ("x" * 80 + "\n") * n
old = bench("./parser_old", data)
new = bench("./parser_new", data)
print(f"{n:>8} old={old:7.4f}s new={new:7.4f}s ratio={new/old:5.2f}")
Run it like this:
python3 compare.py | tee report.csv
And keep the chart. Even a rough ASCII chart works:
size old new ratio
10000 0.020s 0.021s 1.05
30000 0.061s 0.059s 0.97
100000 0.190s 0.148s 0.78
300000 0.620s 0.590s 0.95
1000000 2.100s 2.180s 1.04
Look at the ratios. A real speedup holds across the range. A fake one wobbles. When the new curve only wins at one point, you are measuring noise, not progress.
Decision rules I stole from myself
- Ratio below 0.9 at exactly one size: ignore it.
- Ratio consistently below 1.0 at every size: adopt after correctness review.
- Ratio improves only at the largest size: investigate memory, not CPU.
- Curves that cross: reject. The claimed fix is input-dependent.
These rules are boring. That is the point.
The graph I kept
My parser chart showed the new code winning at 100k rows. It lost at every other size. The median ratio was 0.97. The new parser was not forty percent faster. It was three percent slower on my real workload.
I kept the old parser. I kept the chart. The chart is now part of the pull request trail.
Would I have caught this without the graph? No. I would have merged the patch based on one flattering number.
Limitations
This workflow will not save you everywhere.
Synthetic inputs are not production data. I still replay real request logs before shipping anything. Microbenchmarks also fight with JIT warm-up, CPU frequency scaling, and background noise. I discard the first run and always measure on the same quiet machine.
And a graph only shows speed. It says nothing about correctness. A fast wrong answer is still wrong.
Do not use this approach for I/O-bound systems. If your bottleneck is a database or an external API, CPU time will not reveal it. Use tracing and real traffic for those cases.
The chart answers a narrow question. Ask it before you trust the patch.
A cheap place to run this
MonkeyCode's free server option is enough for this kind of side-by-side test. I still run final numbers locally on a fixed machine. The free tier starts the experiment; it does not replace the discipline.
Check the project page for current terms, because quotas drift. Then run the sweep.
One number is a guess. A graph is evidence. Next time an AI says "40% faster," do not ask for the patch. Ask for the chart.
