How we made service reliability a team job, not an on-call punishment.
The payment-api alert was still firing when we opened the incident channel, and nobody could tell whether customers were actually blocked or whether a dashboard had found another way to ruin breakfast.
SRE means site reliability engineering: a way of running software where the people who build services also set clear expectations for how those services should behave in real use. We measure whether they meet those expectations, respond when they do not, and spend planned time removing repeated sources of failure.
At 08:06, we had a red graph, four Slack replies, and no useful answer to the only question that mattered: could a customer complete a payment? That gap is why we started treating SRE as a working practice rather than a title somebody receives after learning enough Kubernetes nouns.
Start With What A User Can Actually Do
A service can have healthy pods, low CPU use, and a very cheerful deployment status while still failing the one task people came to perform. We learned this with payment-api in April. Its /healthz endpoint returned 200, every container was running, and the PostgreSQL connection pool looked normal. Meanwhile, requests to create a payment were waiting 18 seconds on a call to our fraud provider.
Nobody buying a train ticket cares that our health endpoint is fine.
For SRE work, we begin with a user action: create an account, load a report, upload a document, complete a payment. That action gives us something concrete to measure and protect. It also stops teams from treating infrastructure metrics as proof that an application works.
We wrote down three actions for the payments service:
- A customer can submit a payment.
- A merchant can retrieve its payment status.
- Our nightly settlement worker can send a batch to the bank before 02:00.
Those are not equally valuable. A delayed merchant status page is annoying. A payment request that disappears or hangs is a financial incident. Settlement missing its bank window has a different kind of unpleasantness, usually involving spreadsheets and a person from finance who has every right to be annoyed.
The Google SRE book’s section on service level objectives is useful here because it separates what users experience from whatever machinery happens to sit behind it. We borrowed the idea, then made the wording much plainer in our own service docs.
A user action is the unit we protect; the rest is evidence.
Measure Requests, Not The Mood Of The Cluster
Once we had a useful user action, we needed a number. For payment submission, we count successful HTTP requests and failed HTTP requests. A request is successful if it returns a 2xx response within five seconds. A 500, a timeout, or a response after five seconds counts as bad.
That five-second line is not sacred. We picked it after looking at 30 days of request timing. Most payments completed in 430–900 ms. At 2.5 seconds, support tickets started to correlate with retries. At five seconds, our web client gave up and showed the customer an error. We chose the point where the customer’s experience had already become poor, rather than pretending 400 ms was a promise we could keep during the Monday morning rush.
In SRE terms, the thing measured is a service level indicator, usually shortened to SLI. We call it “the payment success rate” in conversation because acronyms spread quickly enough without our help.
This Prometheus query is close to the one on our dashboard:
sum(rate(http_requests_total{service="payment-api",code=~"2.."}[5m]))
/
sum(rate(http_requests_total{service="payment-api"}[5m]))
The query is deliberately boring. It counts good requests divided by all requests. It does not estimate customer happiness from node load or count pods with names containing payment. Those metrics still have a place when diagnosing a fault, but they are poor promises to make to users.
We also exclude requests made by our synthetic checker from this ratio. Otherwise a healthy checker can keep the graph looking better while ordinary traffic is failing, which is exactly the sort of technical correctness we dislike.
A useful SLI maps a visible user action to a number we can inspect during an incident.
Set A Target You Can Defend In A Meeting
A service level objective, or SLO, is the target for that measurement over a period of time. Ours for payment submission is 99.9% successful requests in a rolling 28-day window.
That means we permit roughly 43 minutes of failed or slow request time per month if every request fails at once. Real faults are messier than that: a partial outage may consume the allowance across thousands of requests in ten minutes. Still, the number gives us a shared boundary.
We tried 99.99% first.
It did not work.
The target sounded suitably serious in the planning document, but it gave payment-api about four minutes and 19 seconds of failure allowance in 30 days. A routine database failover consumed most of that. A deploy that caused a two-minute spike put us in a state where the sensible response was “stop changing anything,” including a fix for the thing that had just failed. We changed it after two months. Nobody has suggested going back.
An SLO should reflect the cost of failure and the cost of making change. For payment submission, 99.9% is defensible because a short failure is costly, but we also deploy fixes and dependency changes every week. For our internal analytics export, 99.5% is plenty. The report is useful at 09:00; it does not need a ceremonial response at 03:00 because one CSV was 12 minutes late.
The target is a negotiated engineering decision, not a number copied from a cloud provider’s brochure.
Treat The Error Budget As Permission To Change
The difference between the target and actual performance is called an error budget. If our target is 99.9%, the remaining 0.1% is the amount of bad service we have agreed can happen in the measurement window.
That phrasing can sound grim. We prefer to see it as permission to take normal engineering risks. Deployments, schema changes, cache migrations, and dependency upgrades all carry some chance of disruption. If we demand zero disruption forever, we get slower changes, stale dependencies, and people quietly avoiding work that needs doing.
At 11:35, during the payment incident, the budget had fallen from 71% remaining to 63%. That gave us room to ship a timeout change and route a small percentage of traffic around the fraud provider. Had we only had 2% left, we would have made a different call: freeze feature releases, use the safer rollback path, and put the reliability work at the top of the sprint.
We keep the policy small:
payment-api:
slo: 99.9
window: 28d
budget_remaining_below: 10%
release_action: "approval from on-call and service owner"
This is not an automatic deployment lock. We tried a hard lock in GitHub Actions last year, and engineers worked around it by classifying urgent changes as operational patches. The YAML survived; the lock did not.
An error budget gives release decisions a number to argue from instead of relying on whichever person is most worried in the call.
Page People Only When They Can Act
An alert should tell an awake person that they need to do something now. Not investigate later. Not admire a graph. Not acknowledge it so that the alert noise reduces by one.
Our old alert set had 146 rules for a cluster of 24 nodes. At 02:40 one Thursday, the on-call engineer received 17 pages for a single Redis failure: node disk pressure, pod restarts, connection errors, cache miss rate, eviction count, and several alerts that had apparently been written by a committee of anxious refrigerators.
We cut that down. The page now fires when the payment SLO is burning quickly enough that we are likely to consume 5% of the monthly budget in one hour. The disk and restart alerts still exist, but they go to a working-hours channel unless they are directly causing user failure.
This uses the “burn rate” approach described in the SRE workbook’s alerting chapter. We like it because it joins urgency to impact. A five-minute blip at 03:00 can wait. A failure eating a month’s allowance before lunch cannot.
The alert notification includes three links: the SLO graph, the relevant deployment history, and the runbook. It does not contain fourteen labels, a container hash, and the hostname of a node that may no longer exist by the time somebody reads it.
The runbook is where we put immediate checks, known failure modes, and safe rollback steps. It is not a wiki page titled “Payments Operational Knowledge” last edited in 2022 by someone who moved teams.
A page earns its place on the rota only when an engineer has a real action available.
Reliability Work Has To Compete For Sprint Time
SRE fails when reliability tasks become “we’ll get to it after the roadmap.” There is always another roadmap item. There is also always another slow query, expiring certificate, or queue consumer that only fails after a long weekend.
We reserve capacity in each service team’s sprint for reliability work. Not a fixed 20% in every case; fixed percentages create odd incentives and fake tickets. Instead, the service owner brings the largest repeat incident, highest-risk dependency, or most worrying budget trend to planning. We ask what would make that problem less likely next month.
For payments, the answer this quarter was not another dashboard. We had plenty of dashboards. We added a circuit breaker around the fraud provider, recorded dependency timing separately, and changed the retry policy so that 400 failed requests do not all retry in the same second.
Our colleague Mina still thinks circuit breakers are a polite name for “giving up early.” She is partly right. We are still arguing about whether failed fraud checks should enter a manual review queue during a provider outage or be rejected immediately. That decision crosses risk, support staffing, and customer trust; no latency graph will settle it for us.
We track reliability work in the same backlog as product work. If it needs a separate spreadsheet to exist, it is already losing.
Words We Use Before They Become Meeting Noise
A short glossary helps because SRE has enough abbreviations to make ordinary incident calls sound like a radio test.
Service level indicator (SLI): The measurement of a user-facing action. For us, payment requests completed successfully within five seconds.
Service level objective (SLO): The target for that measurement over a time window. Our payment submission target is 99.9% over 28 days.
Error budget: The allowed amount of failure implied by the SLO. It is what remains when actual performance is compared with the target.
Burn rate: How quickly the service is spending its error budget. A fast burn rate deserves attention even if the monthly number still looks acceptable.
On-call: The person scheduled to respond when an alert pages. This should rotate, include proper handover, and not quietly become one person’s permanent second job.
Runbook: A practical page for responding to a known alert: where to look, what is safe to change, and when to escalate.
Prometheus has a clear alerting rules reference if you are building alerts from scratch, while Alertmanager’s grouping documentation explains how to avoid receiving seventeen versions of the same bad news.
Words are useful only if the person holding the pager can turn them into a decision.
Keep The First Version Small Enough To Maintain
We have not tried this approach past 40 nodes or across dozens of independently deployed teams. At our size, one platform group and six service teams can still agree on common labels, a dashboard layout, and who owns an alert. Larger organisations will need more structure, and probably more patience than we had at 08:06.
Start with one service that matters, one user action, one measured target, and one page that somebody can act on. Do not wait for a grand reliability programme. We began with payment submission because its failures were visible, expensive, and already waking people up.
By 16:40, the fraud provider had recovered, the error budget had stopped falling, and the incident channel had gone quiet. We still have a node_filesystem_avail_bytes alert that wakes people too often on a particular worker pool. It has been open for 19 days, which is annoyingly longer than the incident that made us write this post.
