A capacity blip at GitHub resulted in an outage that lasted 7 hours and 47 minutes. The infrastructure recovered faster than the clients let it. 😬
What actually happened
GitHub experienced issues on August 17, 2026, from 13:28 to 21:15 UTC. Users were facing problems with Git operations, Actions, Issues, PRs, and Copilot. The error rate for Web and API traffic was around 20%, while for archive and raw-content downloads it was approximately 50%. The root cause of the issue was almost mundane, an Istio sidecar proxy reached its concurrency limit. Here's the nasty part. GitHub's autoscaling monitored the metrics of the host application, not the sidecar's proxy saturation. Therefore, the system detected everything as "normal" while the proxy was overloaded, and no new instances were deployed. Traffic overflowed, causing four HAProxy nodes to exceed their capacity limits, and subsequently degrading the gateway. It was a rough afternoon. However, it's fixable. What made a blip a saga was the following. ## Retries didn't help. They piled on. A hidden bug in the VS Code Copilot extension caused clients to excessively re-request auth tokens from the Copilot Token Service. Backend errors were returned, and the clients then did the "resilient" thing. They tried again. Right away. Over and over. The Token Service typically processes 7,000 to 9,000 requests per second. However, at the time of the incident, it received 70,000 to 100,000 RPS. This means 8 to 14 times the normal amount of traffic, specifically targeting infrastructure that was attempting to recover. In the blog post, GitHub CTO Vlad Fedorov provided a simple explanation: backend errors that "triggered a client-side retry loop that increased traffic during recovery."
They actually made the problem worse. This really frustrates me. It's as if we consider retries to be a measure of resilience. We just add a retry, release the product, and then we don't worry about it. However, an inexperienced retry loop will simply be a self-denial of service attack, albeit with good intentions. Each client that "helpfully" retries is one more member of the botnet that you, yourself, are funding. ## Why the default retry mindset is a trap
The mental model is incorrect here. What people generally perceive a retry to be is a "try again because this time it will most likely work" action. On a large scale, retrying an action on a system that is already overloaded is like adding more burden to it. Failure doesn't result from trying again, but from trying again with no limit and no pause in between. → No budget: clients retry unlimited times, so load has no cap. → No jitter: everyone retries on the same clock, so requests arrive in synchronized waves. → No backoff: retries fire instantly, so a struggling service never gets breathing room. Recovery processes require some slack to function. Simple retries consume all the slack as soon as any failures occur. ## The fix is unglamorous and it works
Firstly, implement budgets to retry the operation. Secondly, use backoff with jitter to delay repeating the operation. This is the pattern. A retry budget limits the number of retries as a percentage of your actual traffic. For example, "the retries should not be more than 10% of the requests." When you reach the limit, retries are no longer performed, and failure is immediately returned. That single rule would've capped GitHub's Token Service surge instead of letting it hit 14x. Backoff with jitter manages the timing. Backoff spreads retries out so a damaged service has a chance to recover. Jitter randomizes the wait time so that clients don't all retry at the same time.Even if jitter is not applied, exponential backoff can still cause thundering herds, since every client waits the same fixed interval and then retries at the exact same moment. All of this is pretty basic stuff that you can find in any good book on distributed systems. Let's face it, resilience is a joint effort between the client and server. A "resilient" client turning a blind eye to the server's suffering is simply a well-mannered adversary. ## The takeaway
Retries should not be seen as a safety net that costs nothing. Unplanned retries can actually put additional strain on your infrastructure when it's already struggling. If you can retry something immediately and for free, you have a bug, not a resilience pattern. Here's a question for you: take a look at your own client code right now. Is there a budget and jitter in your retry logic, or did you simply put the call in a loop and declare it to be "resilient"?
