Last week I ran oasdiff on 13 labelled API change cases I wrote. It caught 8 of the 10 breaking ones (oasdiff v1.33.0, default ERR-level gate) and missed 2. This post is about those 2, because they point at a gap that a spec check alone does not cover: the difference between a change that is breaking in the spec and a change that breaks someone.
Quick context on the cases, so you can judge how much weight to put on this. They are synthetic. One small Orders API, 13 before/after OpenAPI specs, seven made-up consumer services, labels written by me. 10 are breaking changes, 3 are controls (a new optional field, a new endpoint, a renamed path parameter). oasdiff v1.33.0 with --fail-on ERR raised no false alarms. Everything is public: https://github.com/impact-gate/impact-gate-benchmarks
The two it let through
1. Removing an optional response field.
In the spec, a field that is optional in a response is allowed to be absent. So removing it can reasonably be reported as informational. But if a consumer reads that field, for example to show a customer's phone number, the consumer now gets nothing where it used to get a value. The spec said "optional". The consumer treated it as "there".
2. Adding an API key requirement to an operation.
Adding a security requirement is also reported as info by oasdiff, and that is a defensible choice for a diff tool: the spec changed, the API did what it was told. But a consumer that calls the endpoint without a key now gets a 401. For that consumer, the change was breaking on the day it shipped.
Neither report is wrong. A spec diff answers "what changed in the contract". The question a reviewer actually has on a pull request is different: "who will this hurt?"
Why this matters on a pull request
When you review an API change, the spec diff tells you the shape of the change. It does not tell you:
- which other services call this endpoint,
- which of them read the field you are removing,
- which of them call without the header you are about to require.
In a small team you carry that in your head. In a larger one, you find out after the deploy.
What Impact Gate does about this
Impact Gate is a warn-only check that comments on the pull request with the services and code that depend on the endpoint you are changing. Each consumer it lists comes with the file and line where it was found.
There is a live demo on real pull requests you can open yourself: https://impactgate.in . No signup is needed to look around.
I want to be careful about claims. Scoring Impact Gate itself on these 13 cases is not published yet, so I am not claiming it catches the two above. Automated consumer scoring and real-project cases are next*.
* Planned, not available yet.
Where I would like to be wrong
If you disagree with a label, open an issue on the repo. If you have a case where a spec diff passes and a consumer still breaks, I would like to add it. And if you run a different tool against the cases, the runner is TOOL_CMD="your-tool" bash scripts/run.sh.
Question for people who maintain more than one service: how do you find out today who depends on an endpoint before you change it?
