We Broke Production Due to an API Change — Here's How I Fixed It

In a microservices architecture, one small API change can break multiple downstream systems.

I learned this the hard way.

💥 The Problem

We had a backend service that modified an API:

• A field was removed
• Response structure slightly changed

Everything passed:

• Unit tests ✅
• Integration tests ✅
• Code review ✅

But in production:
👉 Another service broke.

Why?

Because we never enforced API contracts.

🔍 Root Cause

We relied on:

• Human review
• Documentation
• Assumptions of backward compatibility

But we were missing:

👉 Automated contract validation

💡 The Solution

I built a simple approach:

1. Store OpenAPI spec (baseline)
2. Compare new spec during CI
3. Fail build if breaking changes detected

⚙️ Implementation

Using a CLI approach:

npx specshield compare old.yaml new.yaml

CI Integration:

- name: Check API breaking changes
  run: npx specshield compare

🚨 What counts as breaking?

• Removing endpoints
• Removing fields
• Changing types
• Making optional → required

🧠 Key Learnings

• API contracts should be treated like code
• CI is the best place to enforce safety
• Even small changes can have big impact

🔮 Future Improvements

• Consumer-driven contracts (Pact-style)
• Better diff visualization
• Integration with GitHub PR checks

🤔 How do you handle this?

• Do you use OpenAPI validation?
• Pact?
• Custom scripts?

Curious to learn how others solve this problem.

https://specshield.io/