Performance gets worse and worse as patches are added. None of the specific PRs contribute 200ms of latency, one contributes 3ms, another 5ms, and one 8ms, as someone implemented a "quick" N+1 query behind a feature flag that then became the default. Six months later you're 150ms slower than before, nobody can point to the commit that made you so slow, and a complete profiling investigation is the only way to determine where you are going. Don't fix it when you profile it, fix it when you regress it.
Treat perf budgets like any other CI gate
A performance budget is a number that your team agrees that a code path will not exceed, such as endpoint latency at p99, memory footprint of a service, bundle size of a frontend build. As with the point tests and lint, the budget should be applied pre-merge, in the CI environment and to a number which is tracked in version control along with the code rather than in a dashboard that no one checks until it turns red.
# .github/workflows/perf-budget.yml
name: perf-budget
on: [pull_request]
jobs:
bench:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: go test -bench=. -benchtime=3x -run=^$ ./... > current.txt
- run: git show origin/main:bench-baseline.txt > baseline.txt
- run: benchstat baseline.txt current.txt
- name: fail on regression over budget
run: |
python3 scripts/check_budget.py \
--baseline baseline.txt --current current.txt \
--max-regression-pct 5
The statistical comparison is done by the benchstat, part of Go's golang.org/x/perf suite of tools, which takes into account noise so you are not failing builds on a 1.2% fluctuation. The budget check script is the script that you create yourself, and it should fail loudly with the name of the specific benchmark, and the percentage regression.
Store the baseline in the repo, not in a dashboard
The big design decision is to place bench-baseline.txt (or equivalent) into the repository and only update it purposefully, as a step taken as a deliberate and reviewed action (usually in conjunction with a PR that deliberately sacrifices performance for some other gain, such as a feature that's worth it). This makes performance budget changes an accountable, reviewable change like any other code change:
# When a PR legitimately needs to update the budget:
go test -bench=. -benchtime=5x -run=^$ ./... > bench-baseline.txt
git add bench-baseline.txt
git commit -m "perf: raise checkout latency budget 12ms -> 18ms for fraud check v2
Trading latency for the new fraud detection pass. Approved in
#4021 — expected fraud loss reduction outweighs the latency cost.
See perf/2026-08-fraud-check-budget.md for the analysis."
Now if you type in git blame bench-baseline.txt, you get a real history of all the deliberate performance compromises that your team has made and the rationale for each, as documented in the commit where it was made. This is a much better place to get the truth than an 8-month-old Slack thread that nobody can find.
Scope budgets to what actually predicts user-facing pain
No, don't attempt to budget everything – you will become alert fatigued and the team will tune out the CI check – and that is not the point. For each of those few code paths with actual cost, pick it.For the handful of code paths that have actual cost, pick them: the "hot endpoints", the "critical rendering path", the necessary functions that show up wide on your production flamegraphs (see: latency archaeology). If it's an admin endpoint that you're not seeing very often, it's noise, if it's on your checkout path, or on your auth middleware, it's signal.
# scripts/check_budget.py — core logic
def check(baseline: dict, current: dict, max_pct: float) -> list[str]:
failures = []
for name, base_ns in baseline.items():
if name not in current:
continue
cur_ns = current[name]
pct = (cur_ns - base_ns) / base_ns * 100
if pct > max_pct:
failures.append(
f"{name}: {base_ns}ns -> {cur_ns}ns (+{pct:.1f}%, budget {max_pct}%)"
)
return failures
Make the regression PR-blocking, not merge-blocking after the fact
The whole point of this is that this error is caught at the commit that introduced it, as the author still had all the context and can resolve it in 5 minutes instead of 5 hours, next quarter, doing a lot of archaeological work. If a nightly job recovers the same regression a week after the three subsequent PRs, it has already lost most of its value, you've got bisect to do again. Instead, perform the budget check as a status check that is required to be run on the PR itself, not as a separate scheduled job.
This is a real cost; there are costs to benchmarks for them to be stable enough not to flake, and there are costs to having someone with the updating budgets when they need to move. The other choice is the kind of situation that most teams are in right now: performance gradually sneaks up on them for months, then one person spends a week doing "latency archaeology" to discover it was 6 totally unrelated commits, all of which were individually justifiable, and it all added up to the problem no one signed off on.
