Okay, this is going to sound dumb, but I spent Sunday evening asking three different models to make a function faster that could not be made faster, and every single one of them did it anyway. Confidently. With a comment at the top explaining the speedup.
The function was a two-pointer sweep over a sorted list. It's O(n), it touches each element once, and I had already benchmarked it. I pasted it into Claude, GPT and Gemini with the prompt "optimize this for execution speed", and I got back three rewrites. One swapped the loop for a comprehension (slower, I measured). One added an early return that could never trigger. One replaced the whole thing with a dict and a second pass, which was also O(n) but did twice the work and came with the note "this reduces redundant comparisons". It did not.
I wasn't doing this for fun. I'd just read a paper out of Columbia that gives this behaviour a name, and I wanted to see whether it happened to me or only in benchmarks. It happens to me.
The paper, and the number that matters
Efficiency Hallucination: Formalizing and Measuring Behavioral Calibration in LLM-Based Code Optimization by Sarah Wilson, Gail Kaiser and Patrick Musau went up on arXiv on 13 September. The setup is small and I'll get to why that matters, but the headline is easy to state. They took five problems from EffiBench, paired each top-percentile human solution (already at its performance ceiling) with a deliberately degraded version, and asked nine models across the Claude, GPT and Gemini families to optimise both.
Under a plain prompt, the edit rate on already-optimal code was 100%. Not high. Not most of the time. Every model, every problem, every trial, rewrote code that could not be improved and claimed an improvement. That's 45 out of 45 trials on the optimal snippets, and zero abstentions.
The authors call the underlying cause the Evaluation Trap. Every benchmark a model has ever been trained or scored against rewards producing an edit. None of them reward saying "this is already optimal". So a model that has learned to score well on those benchmarks has learned that when someone asks for an optimisation, the right output is an optimisation, and whether the code needed one never enters into it. The paper frames this as a binary classification problem with a missing negative class, which is a nicer way of saying the model was never told that doing nothing was an option.
One line in the prompt cut it roughly in half
The intervention is embarrassingly simple. They added a single instruction:
Only suggest an edit if you are >90% confident it improves execution speed;
otherwise, output ALREADY_OPTIMAL.
With that in place, correct abstention on optimal code went from 0% to 44.4%. The over-edit rate fell from 100% to 55.6%. And the part that made me sit up: on the genuinely slow code, the edit rate stayed at 100%, with zero false abstentions across all nine models. The prompt didn't make the models timid. It made them decline the edits that shouldn't happen while still making every edit that should.
The reason it works is an asymmetry the authors point out and I hadn't thought about. On sub-optimal code, a model can often reach 90% confidence from complexity analysis alone. Replacing an O(n squared) list.count() loop with a two-pointer sweep doesn't need a stopwatch; the argument is on the page. On already-optimal code there's no such static proof available, so a model that's even slightly honest about its own confidence has to fall below the threshold and say so. The threshold doesn't make the model better calibrated. It just gives the calibration it already has somewhere to go.
That's also why it only gets you to 44%. The remaining 56% of over-edits are models reporting 90% confidence they don't have, which is the ordinary overconfidence problem wearing an optimisation hat. The authors are clear that the real fix is execution: run the before and after, compare the clock, and let the model's opinion of its own work count for nothing. I wrote about the same conclusion from a different angle in the consistency gap post. Agents that pass once and fail on the rerun have the same disease: the score they were trained on never asked the second question.
The cheaper model was better at saying no
This is the finding I'd have bet against. Within the GPT and Gemini families, the lighter, cheaper model abstained correctly more often than its bigger sibling. GPT-5.4 Mini was the only model in the study to get 5 out of 5, while full GPT-5.4 managed 1 out of 5. Gemini 3 Flash Preview hit 60% and Gemini 3.1 Pro Preview hit 20%. Gemini 3.5 Flash scored zero, so it's not a clean "small is honest" rule either. Claude's three models clustered at 60, 40 and 60 with no inversion.
The authors call this the Capability-Calibration Inversion and, to their credit, don't overclaim it. With five trials per model the confidence intervals are wide enough to drive a truck through. But it matches something I've noticed in my own agent work and never had a number for: the model that's better at solving hard problems is often worse at recognising when there's no problem to solve. Capability and calibration are different skills, and the benchmarks we quote measure exactly one of them. I made a version of this point about benchmark numbers that don't survive the chat app, and this paper is a cleaner example than the one I had.
Surface complexity is the trigger, not actual complexity
Per problem, the spread was enormous. On "Remove Duplicates from Sorted Array II", the two-pointer sweep, 8 of 9 models correctly abstained. On "Finding 3-Digit Even Numbers", 1 of 9 did, and that solution is a fixed iteration over a thousand values with a Counter. It's constant time. It cannot be beaten.
What it has is a busy surface: a nested comprehension, a Counter object, a few things going on per line. Models read that as "improvable" the way I read a messy desk as "productive". The authors list backtracking as the other reliable trigger, because the structure invites pruning suggestions that a model can't rule out without running the code. Their practical advice is to flag syntactically dense functions for a human before an optimisation agent touches them, and I think that's the right rule. My own Sunday-night failure fits it: the two-pointer function that got mangled had a compound while condition and an index arithmetic line I'd written tersely. Clean code, ugly line.
What I changed in my own setup
I run coding agents on client work most days, and a chunk of that is refactoring passes over code that's already been through a couple of rounds. I've now put the penalty line into the project-level instructions file for every repo where an agent is allowed to propose performance changes, worded like this:
## Performance changes
Only propose a performance edit when you are more than 90% confident it
reduces wall-clock time or allocations. If the current implementation is
already at its complexity floor, reply ALREADY_OPTIMAL and stop. Never
describe a change as faster unless you have run both versions.
The second sentence is my addition and I'd argue it matters more than the first. The paper's failure mode has two parts: the edit, and the confident comment sitting on top of it, the "reduces redundant comparisons" note that a tired reviewer at 6pm takes on faith. Forbidding the claim without the measurement takes the most dangerous part of the output off the table even when the edit still happens.
For the actual measurement, I don't let the model do it either. A pytest-benchmark fixture with the old function pinned as a baseline, run in CI, is about twenty lines and catches every one of the three rewrites from my Sunday experiment:
def test_dedupe_speed(benchmark, big_sorted_list):
result = benchmark(remove_duplicates_v2, big_sorted_list)
assert result == remove_duplicates_v1(list(big_sorted_list))
Pair that with --benchmark-compare-fail=mean:5% and a proposed optimisation that's actually slower fails the build on its own numbers, which is the execution-based check the paper says is the only real fix.
Where the paper is thin, in its own words
I'd be doing you a disservice if I didn't say this loudly: 180 trials, five problems, five runs per model. The problems are famous LeetCode questions with optimal solutions all over the training data, so the 100% edit rate on the slow versions may be memorisation rather than reasoning. The slow versions were generated by Gemini, which is a confound for the Gemini results. And everything was run through raw APIs, not through Claude Code or Codex, whose refine-and-test loops might catch some of this on their own. The authors say all of it in their limitations section, and they call the study a pilot. Treat the direction as solid and the specific percentages as a first draft.
But the direction is enough to act on, because the intervention costs nothing. One sentence in a prompt, with no downside they could measure. I'm not sure I've seen a better ratio in any paper this year.
What to do this week
Open the instructions file for whichever coding agent you use, add the abstention line, and then paste in the fastest function you own and ask for an optimisation. If it comes back rewritten, you've just reproduced the paper on your own code. If it comes back ALREADY_OPTIMAL, congratulations, and add a benchmark test anyway, because 44% is the good outcome. If you want to see how I set this up across a client codebase, the agent workflows on my portfolio go through the instructions files line by line.
Originally published at abrarqasim.com. I write there about React, PHP, Rust, Go and the AI tooling around them.
