Retries, Budgets & Retry Storms

The timeout told you when to stop waiting. The obvious next move is to try again — and Calling Services Safely: Timeouts, Retries & Backoff already taught you to do it well: only retry transient errors, never a 4xx, only idempotent operations, always with backoff and jitter, and put a circuit breaker in front. Those are the moves for one client.

This lesson is what those moves do to a fleet. A retry is the strangest tool in the reliability kit: it makes a single client more reliable right up until the dependency is struggling, and then it inverts — the exact mechanism you added to survive a blip becomes the traffic that turns a blip into an outage. By the end you'll understand the two things that separate a retry that saves you from one that kills you: a budget that caps the total, and the tipping point past which a system stops being able to heal itself at all.

Start with the multiplication, because it's worse than most people guess. Retries don't add load; they multiply it, and down a call tree they multiply at every level.

One user clicks once. The edge calls the service and, configured to retry three times, can send three. Each of those reaches a service that also retries three times, so nine requests can reach the database. The database client retries three times too, and twenty-seven calls land on the disk. Three layers, three attempts each, and a single healthy click has become twenty-seven calls at the bottom — and that's the calm number, before anything is failing. The instant the bottom slows down, every one of those twenty-seven starts timing out and retrying, and the multiplication compounds.

This is why the first rule of fleet-scale retries is counterintuitive: retry at exactly one layer. Pick the edge, or a single deep hop, and make every other layer pass failures straight through. Retrying "everywhere, to be safe" is how you build a 27x amplifier and point it at your most fragile dependency. One layer retries; the rest report.

There's a sharper version of the amplification that hides inside the previous lesson's lesson. When a call times out, you learned in Timeout Hierarchies that the work often doesn't stop — the timeout is your patience running out, not the server's work stopping. The request is applied-or-lost, and the server may still be grinding on it.

Now retry it. You have not replaced the first call; you've added a second one on top of it. For a moment the struggling service is doing the work twice, and you did that precisely when it was already too slow to answer once. A retry-on-timeout against an overloaded service doesn't recover the request — it doubles the load at the worst possible instant.

This is where the previous lesson's loose thread gets tied. A budget can be spent per try — each attempt gets the full timeout, so two attempts can cost twice the wall-clock and run twice concurrently — or per request — all attempts share one deadline, so the retry inherits only what's left. Per-request is almost always what you want: it means a retry can't push a doomed request past the budget the user was promised, and it means a slow first attempt simply leaves no room for a second. The deadline that rides the request, from last lesson, is what makes per-request retries possible at all.

Backoff and jitter, from the earlier lesson, shape when retries land — they spread the wall of traffic into a smooth curve. What they cannot do is cap how much. A thousand clients retrying politely, with perfect jitter, still send a thousand extra requests. Smoothing the spike is not the same as shrinking it.

The tool that caps the total is a retry budget, and it is refreshingly concrete. Model it as a token bucket: ordinary traffic adds tokens, each retry spends one, and the bucket is sized so that retries can never exceed a small fraction of the base load. A retry that finds the bucket empty isn't delayed — it's refused. The client gives up now instead of throwing one more log on the fire.

The striking thing is how sharply the industry has converged on the same number. Envoy's retry budget defaults to 20% of active requests. Finagle's defaults to 20%, plus a floor of ten retries per second, on a token bucket whose credits expire after ten seconds. gRPC's retry throttling pauses retries entirely once its token count falls below half. Google's SRE practice describes the client-side version as adaptive throttling: a client watching its own recent rejections starts declining its own requests, and under heavy overload settles into rejecting roughly one retry for every request the backend actually accepts — a decision made locally, with no coordination. Different companies, different code, one rule: retries are a small, bounded fraction of traffic, never an open tap.

A budget changes the shape of failure. Without one, a struggling dependency invites unbounded amplification and the storm feeds itself. With one, the extra load is capped at twenty-odd percent no matter how bad things get — enough to paper over real blips, not enough to convert a stumble into a stampede.

Now the deep idea, the one that turns a bad afternoon into a multi-hour outage and shows up in the post-mortems of the biggest cloud providers on earth. It has a name from a 2021 paper by Nathan Bronson and colleagues: metastable failure.

Picture a service running comfortably under its capacity line. Something transient shoves it over — a deploy, a brief dependency hiccup, a traffic spike lasting seconds. Requests start failing. The clients, doing exactly what you told them, retry. And here is the trap: those retries are now themselves the overload. The original trigger passes — the deploy finishes, the spike subsides — and the service still can't recover, because the load holding it down is no longer the trigger. It's the retries, which exist only because the service is failing, which continues only because of the retries. A self-sustaining loop. The system has settled into a second stable state: not healthy, not crashed, but wedged — degraded, and staying there.

This is why the two most natural instincts both fail. "Wait for it to recover" fails because the sustaining loop has no reason to stop on its own. "Add capacity" fails, or barely helps, because the retries scale with the fleet — more servers invite more clients whose retries fill the new capacity too. A metastable system doesn't have a shortage you can buy your way out of; it has a loop you have to break.

And breaking the loop is exactly the toolkit of this section. Shed load so the service serves a fraction cleanly instead of failing all of it. Open a circuit breaker so clients stop calling for a beat and the queue drains. Cap retries with a budget so the sustaining effect can never grow past a fraction of real traffic in the first place. Each one attacks the same thing: the feedback that keeps a recovered cause from producing a recovered system. The trigger is not the disease — the loop is.

Below is a service with a capacity line, a retry loop, and a budget you control. Leave it healthy first and watch offered and served load move together. Then hit trigger a blip — a spike over capacity for a couple of seconds — and release it. With the budget off, sit and watch: the trigger is long gone, and the service never comes back. Offered load stays pinned above the line, held there entirely by the retries of the requests it can't serve. That flat, stuck, above-the-line state with the lamp reading METASTABLE is the outage, live.

Now break the loop. Turn the retry budget down to twenty percent, or shed load, and watch the effective load fall back under the capacity line and the lamp flip to HEALED — with the trigger never having returned. Then try to heal it the tempting ways instead: wait longer, or add capacity, and watch neither one work. The loop, not the trigger, is what you have to cut.

• A retry inverts under load. It makes one client more reliable until the dependency struggles, then it becomes the traffic that finishes the job. The client-side hygiene from Calling Services Safely: Timeouts, Retries & Backoff is necessary and not sufficient.
• Amplification is multiplicative: retry at one layer, never at every layer. Three layers times three attempts is twenty-seven calls from one click, before anything is even failing.
• Retry-on-timeout doubles the load on an already-slow service, because the first attempt is applied-or-lost and often still running. Spend the budget per request, not per try.
• Only a budget caps the total. Backoff and jitter smooth the spike; a token-bucket budget bounds it — and the whole industry lands near twenty percent of traffic.
• Metastable failure is a loop, not a shortage. A transient trigger plus retries as the sustaining effect equals a collapse that outlives its cause. You don't wait it out or buy your way out; you break the feedback: shed, trip, budget.

A budget bounds how much you retry, but it never answers a blunter question: what if the dependency isn't slow, but dead? Retrying a corpse at twenty percent is still retrying a corpse — polite, bounded, and completely pointless, and every one of those calls still costs you a thread and a timeout. The tool that notices a service has stopped answering and stops calling it entirely, then quietly checks whether it's back, is next: Circuit Breakers.