A pull request can be conflict-free, have green CI, and still lack evidence that the final merged state was ever checked.
That is the narrow problem behind merge-proof, a small open-source Git / GitHub Action tool I just published.
This is not a claim that AI-authored code is broken. It is not a claim that GitHub CI is broken. It is not a claim that a flagged PR contains a bug.
It is only about evidence.
The strongest objection first
If your repository uses strict required status checks correctly, and especially if it uses a properly configured GitHub merge queue, that is the stronger answer to base drift.
A merge queue tests the combined state before merge. For the specific problem of "did this PR get validated against the base it is actually merging into?", that is better than a diagnostic tool after the fact.
merge-proof is narrower. It is useful where that stronger enforcement is absent, incomplete, advisory, or not easy to inspect. It surfaces the evidence gap: which base commit, which candidate commit, which overlapping files, and which protected-path categories made the result unproven.
If your conclusion is "GitHub already solves the valuable part of this for my repo," that is useful feedback, and I would rather hear it than not.
The merge-state evidence problem
A common PR path looks like this:
- A branch is cut from
mainat commit A. - CI runs on the candidate branch and passes.
-
mainmoves to commit B while the PR is open. - The PR still merges cleanly.
- The final merged state is now B plus the PR.
The clean merge only tells you Git could combine the text. It does not prove that CI, review, or any other validation checked that exact combination.
This matters most when the base changed in files the PR also changed. There may be no conflict, but the combined behavior was not the state represented by the earlier green check.
What merge-proof checks today
merge-proof currently implements two blocking checks.
BASE_DRIFT_UNVERIFIED — the base advanced after the candidate diverged, and files changed by the candidate were also changed on the base during that interval.
PROTECTED_BOUNDARY — the candidate changes paths classified as expensive to undo if merged without explicit evidence: migrations, schemas, auth, secrets, billing, or access-policy / RLS / RBAC areas.
It also reports STALE_BASE as advisory only, and reports CI and deployment configuration changes as an advisory ci-deploy category. Neither advisory changes the verdict.
The tool runs on local Git state. It has zero npm dependencies. It makes no network calls, no model calls, no API calls, and has no telemetry. In shallow clones it fails closed instead of reporting confidence from history it could not see.
One disclosure: the GitHub Action wrapper's only third-party dependency is actions/github-script@v7, a GitHub-owned action, used solely to post the optional sticky PR comment. The verifier itself has none.
What merge-proof does not check
This release does not read CI results, so it cannot establish CI_RAN_ON_FINAL_HEAD.
It does not read reviews or approvals.
It does not verify that a candidate is durable on a remote.
It does not compare a PR against a declared scope.
It is not a bug detector, security scanner, code reviewer, human-approval verifier, or correctness proof. A VERIFIED result only means the implemented checks did not find their specific evidence gaps.
The study
We ran merge-proof against 2,037 agent-authored PRs across 32 public repositories. 24.1% returned NOT_PROVEN under the implemented checks.
Immediately important: NOT_PROVEN does not mean broken or buggy. It means the repository does not contain evidence that the candidate was validated against the state it merged into. No conclusion about code correctness follows from that verdict. This is a prompt to look, not an accusation.
The exact result:
- 2,037 usable PRs
- 42 unresolvable PRs excluded from all rates
- 490
NOT_PROVEN, or 24.1% - 1,547
VERIFIED, or 75.9% - 0
FAIL - 410 base-drift findings, or 20.1%
- 109 protected-path findings, or 5.4%
- 29 PRs triggered both conditions
The study used deterministic Git data. Agent-authored PRs were identified from commit trailers, not model judgment. The scan walked squash merges on the default branch's first-parent history, used the squash commit's first parent as the base-at-merge, fetched the PR head directly, and ran the same analyzer that ships in the package — not a separate research implementation.
It is reproducible from the repository:
bash
node study/collect.js --repos study/repos.txt --out study-data.json --max-prs 80 --prune true
node study/summarize.js study-data.json
