The slide always looks the same. Pilot spend: $80K. Velocity: up 9%. Recommendation: expand to the full org. I have sat through enough of these readouts, on both sides of the table, to recite the speaker notes from memory. The room approves it, because $80K buying a 9% faster engineering org of any size is obviously a deal.
The math on the slide is fine. The problem is a line item that never made it onto the slide, and it is the most expensive input in the building.
Where the $80K Went
Run the composite pilot. Fifty seats of an AI coding assistant at roughly $20 a head is about $12K a year. Agent and API tokens for the teams that went past autocomplete, call it $3K a month once real workloads land — I've published my own bill, so I know what this curve looks like. Add the integration sprint, the eval work, and the enablement time nobody bills to the pilot, and $80K is a fair all-in number for a mid-size org's first serious year.
Every dollar of that is metered, invoiced, and visible. Seats show up on a purchase order. Tokens show up on a usage dashboard. Finance can read the pilot's cost to the penny, which is exactly why the readout feels rigorous.
Now the return side. A 9% velocity gain across a 12-engineer group is roughly one extra engineer's worth of throughput, and a loaded engineer runs $250K, which is the number that dwarfs every token bill in sight. So the slide implies $80K bought about $270K of capacity. A 3.4x return, approved before the coffee gets cold.
The Line Item That Never Shows Up
Here is what the slide does not carry: the review and verification attention the pilot consumed, priced at senior-engineer rates.
AI-assisted pull requests change shape. Diffs get bigger. Volume goes up. And the failure mode shifts from obviously-broken to plausible-but-wrong, which is the expensive kind, because plausible-but-wrong has to be caught by a human who understands the system. Suppose each engineer ships five AI-assisted PRs a week and each one takes a reviewer just twenty extra minutes. Across 12 engineers that is 20 hours a week of review time, half a full-time engineer, roughly $125K a year. And it does not land evenly: it lands on the two or three senior people qualified to catch the plausible-but-wrong category, the same scarce attention that was already the bottleneck.
Take the $270K gain, subtract $80K in tool spend and $125K in unpriced senior attention, and the triumphant 3.4x collapses to about 1.2x, before counting rework. That might still be worth doing. But nobody in the room approved that deal, because nobody saw it.
The reason nobody saw it is structural, and it is worth saying plainly: finance manages what is metered. Seats and tokens arrive as invoices. Attention arrives as nothing at all. It is smeared invisibly across a salary line that looks identical whether your seniors spent the quarter building or spent it correcting a machine's confident guesses.
The Evidence Says the Slide Flatters Itself
The 9% is usually self-reported or lightly measured, and self-report runs hot. In METR's randomized study of experienced open-source developers, participants estimated AI made them 20% faster while the measured result was 19% slower. The perception gap is the mechanism that hides the missing line item: the tool feels fast because typing is fast, while the slow part — verifying, correcting, re-prompting — doesn't register as "using the tool."
The vendor numbers run hotter still. GitHub's own study reported 55% faster task completion, a first-party figure measured on a greenfield toy task, and it should be discounted accordingly. Meanwhile DORA's research keeps finding the bill on the other side of the ledger: as AI adoption rises, software delivery stability dips. Faster generation, wobblier delivery. That wobble is rework, and rework is more senior attention, unpriced.
To be fair to the other side of the argument: METR's result is one population on one kind of task, not a universal law, and pilots built on scoped tasks with an explicit verification budget genuinely do clear the honest bar. The point is narrower and harder to dodge. A pilot that measured only what was invoiced has not measured its return. It has measured its receipts.
The Five-Line Version
The fix costs nothing but honesty. Here is the pilot P&L with all the lines on it:
AI coding pilot P&L
+ Velocity delta
measured (cycle time,
merged scope), never
surveyed
- Seats + tokens
the invoice (the only
line the readout had)
- Review-hours delta
PR review time before
vs. during, x reviewer
cost
- Rework delta
change-failure/rollback
rate, before vs. during
- Comprehension debt
merged code nobody on
the team can explain, %
The first two lines exist in every pilot. The last three exist in almost none, and they are all measurable with tooling you already run: your git host timestamps every review, your incident tracker already counts rollbacks, and the comprehension question takes one uncomfortable team meeting.
If the pilot still clears the bar with five lines on the page, expand it with confidence — mine does, and that's precisely why I keep publishing the full bill instead of the flattering half. A tool that survives honest accounting is a tool worth scaling.
Ask for the Line
The 9% on the slide is neither a lie nor a result. It is an unfinished calculation, presented with the confidence of a finished one, to a room that is being asked to multiply it by the whole org.
So ask the one question that finishes it: show me the review-hours delta. If the answer is a number, you are looking at a rare, well-run pilot, and you should probably fund it. If the answer is a pause, then the 9% was never a measurement. It was a survey wearing one, and the most expensive people in the building are quietly paying the difference.
