I ran a contract check against the Swagger Petstore. Here is what came back.

Most API bugs I care about are not crashes. They are small lies between the spec and the live API. A field that went missing. A status code nobody documented. A body that is not what the spec says it is.

I built SpecSentinel to catch those. It is a small command line tool. You give it an OpenAPI spec and the address of a running API. It sends GET requests, compares every answer with the spec and ends with an exit code: 0 for a match, 1 for drift, 2 if the check could not be done. That makes it easy to use as a step in CI.

To see what it does on something real, I pointed it at the public Swagger Petstore, the demo API from the OpenAPI world.

specsentinel https://petstore3.swagger.io/api/v3/openapi.json --url https://petstore3.swagger.io/api/v3

This is the real output, shortened:

GET /pet/findByStatus       200  OK
GET /pet/findByTags         500  OK
    warning SERVER_ERROR             status
            server error 500 is only covered by the default response
GET /user/login             200  DRIFT
    error   INVALID_JSON             body
            response is declared as JSON but the body is not valid JSON
GET /user/{username}        200  OK

8 checked, 1 with drift, 0 skipped, 0 failed
Result: DRIFT (exit code 1)

Two things came out of it.

GET /user/login is documented as JSON and answers with Content-Type: application/json, but the body is plain text. A client that trusts the spec and parses the answer as JSON will fail. That is exactly the kind of drift I built the tool for.
Three endpoints answered with a 500. The spec only covers server errors through its default response, so SpecSentinel reports them as warnings, not as errors.

A fair note: the Petstore is a shared demo server, so results can change from day to day. This is not a complaint about the Petstore. It is a real spec and a real server, which makes it a good test.

What the tool does not do yet: it checks GET requests and JSON responses only. I also ran the schema checks against the public specs of the Petstore, GitHub and Stripe (920 GET operations). For every response I generated a conforming example from its schema and fed it through the checker. It produced no findings and no crashes. That shows the checker does not raise false alarms on large real specs. It does not prove that every kind of drift is caught, and it is no replacement for running it against your own API.

Version 0.2.0 adds warnings for string formats, lengths, number ranges, patterns and required response headers, JSON output for other tools, and a one line GitHub Action.

Try it:

pip install specsentinel

The code is on GitHub: https://github.com/b6bs62fhys-jpg/specsentinel

If you run it on your own API and it says something wrong, or misses something it should have caught, I want to hear about it. That is the feedback that makes the tool better.