Cascading Failures

Tail Latency: Living at p99 ended with a threat. A slow response is not just slow; while it hangs it is holding a thread, a connection, a slot in a pool, and a fat tail backs up and fills those pools. This lesson is what happens when that backing-up doesn't stop. It is the failure this entire section has been circling, the one every earlier tool was quietly built to prevent: the cascading failure, where one component's trouble spreads to its neighbors, and theirs to their neighbors, until a large part of the system is down.

The shape, in Michael Nygard's words, is that a crack in one layer triggers a crack in the layer that calls it. That is the whole difference between an incident and a catastrophe. A component failing is normal and survivable; you have spent six lessons learning to contain it. A component failing and taking everything that depends on it down with it is an outage that starts with one slow database and ends with a status page full of red. The frightening part, which the rest of this lesson is about, is that cascades are not rare freak events. They are the default behavior of a system with no defenses. Left alone, failure spreads. The work is in stopping it.

Start with the mechanism, because it is more physical than it sounds. Picture a chain of services: the user hits a root, which calls a service, which calls another, which reads from a database. Now the database gets slow, from any of the ordinary causes in Tail Latency: Living at p99.

Watch what its caller does. Every request that caller sends to the database now holds a worker thread and a connection for the whole slow duration, waiting. Those are bounded resources, so they run out: the caller's thread pool fills with requests all stuck waiting on the slow database, and with no free threads it can no longer answer the service above it. So that service's threads start piling up waiting on this one, and its pool fills, and the exhaustion travels upward, from the leaf toward the root, one layer at a time. The user-facing service dies last, killed by a database three hops away that it barely thinks about. Resource exhaustion propagates upstream, and that is how a single slow backend becomes a total site outage.

The resources don't fail politely one at a time, either; they fail together, in a knot. Google's SRE book traces a canonical version: a frontend with badly tuned garbage collection runs low on CPU under load, which slows every request, which means more requests are in flight at once, which uses more memory, which shrinks the cache, which raises the cache-miss rate, which pushes more load onto the backend, which runs out of its threads, whose health checks then fail, at which point the cascade is fully under way. CPU, memory, threads, connections, file descriptors: exhaust any one and it drags the others down with it. There is rarely a single clean cause to point at, only a chain that pulled itself tight.

If cascades only climbed, they would be bad but bounded. What makes them uniquely terrifying is that they feed themselves: a cascade is a positive feedback loop, and once it starts, the failure creates the exact conditions for more failure. Two engines drive the loop.

The first is retry amplification, from Retries, Budgets & Retry Storms. When requests to an overloaded service start failing, its clients retry, and the retries are additional load landing precisely when there is none to spare. Google's book puts a number on how ugly it gets when retries stack across layers: four retry attempts at each of three layers is not twelve attempts, it is four times four times four, sixty-four attempts hammering the database at the bottom. The failure manufactures its own traffic.

The second engine is load redistribution, and it is the one that turns a stumble into a rout. In a cluster, when one server dies its share of traffic is spread across the survivors. But the survivors were already near their limit, so the extra load tips them over, and now they die, and their traffic redistributes onto even fewer machines, each of which now gets an even bigger share. Each death makes the next death more likely. The fleet crash-loops its way to zero, and the death of the first server is what killed the second.

And here is the property that makes a cascade so hard to end, the one worth burning into memory: hysteresis. The loop sustains itself far below the load that started it. Google gives the stark version: a fleet that was perfectly healthy at 10,000 requests a second, and that tipped into a cascade at 11,000, will not recover when you drop the load back to 9,000. It may not recover at 5,000. You might have to drop all the way to 1,000 before the crashing stops, because crashed servers keep restarting straight into a wall of queued work and dying again before they can stand up. This is the metastable failure from Retries, Budgets & Retry Storms at the scale of a whole fleet. The trigger was transient; the loop is not. Removing the cause does not remove the cascade — which means your instinct in the incident, to undo whatever you just changed, is often not enough.

This is a thing you have to watch move. Below is a four-tier call chain — a user-facing root, two services, and a database — with requests flowing through it and every defense in your hands.

Make the database slow and watch the failure climb: the database's caller fills its pool waiting, reddens, and stops answering; the tier above it reddens next; and the root goes down, which is the moment real users start seeing errors. Turn on client retries and watch the cascade arrive faster and burn hotter. Then flip on the tools one at a time and watch each one cut the chain at a different link: a timeout ends the endless wait so the hold can't back up; a circuit breaker stops calling the dead database at all; a bulkhead keeps the slowness sealed in one compartment; load shedding keeps the root alive at capacity instead of dying. The lesson lands the moment you notice the failure always climbs to exactly the first unprotected link and stops there. Leave a gap anywhere in the chain, and the cascade finds it.

Now the payoff, and the reason this lesson sits at the end of the section rather than the start. Every tool you have met was, in truth, a way to cut one specific link of the cascade chain. Lay them out against the failure and the design of the whole section becomes visible.

Timeouts, from Timeout Hierarchies, cap how long a caller will wait on a slow dependency, so a slow reply can't hold a resource forever and back up the graph. Retry budgets, from Retries, Budgets & Retry Storms, cap the amplification engine so failures can't manufacture a flood. Circuit breakers, from Circuit Breakers, cut the propagating edge outright by refusing to call a dependency that's already down. Bulkheads, from Bulkheads, keep one dependency's exhaustion inside its own bounded pool, so it can't drain the resources its healthy neighbors share. Backpressure, from Backpressure, pushes the overload back up the line as a signal instead of letting it pile into an unbounded queue. Load shedding, from Load Shedding & Graceful Degradation, keeps an overloaded node alive and serving at capacity instead of crashing and dumping its load onto its neighbors — which is precisely how you break the redistribution spiral. And the tail-tolerance of Tail Latency: Living at p99 attacks the very first link, keeping one slow reply from holding a resource in the first place.

The lesson in that list is defense in depth. No single tool prevents a cascade, because a cascade is opportunistic: it will find the one link you left unprotected and climb through it. You need the timeout and the breaker and the bulkhead and the shedding, because each covers a failure the others don't. Two more defenses live outside the request path. Capacity headroom: if a cluster breaks at 5,000 requests a second, you run enough clusters that you can lose two and still serve peak, because a system run at its tipping point is one hiccup from the spiral. And a caution that adding capacity is not always safe: in the 2020 outage where Amazon's Kinesis service went down for most of a day, the trigger was adding servers to a fleet whose members each had to talk to every other member, so more servers meant more threads on every existing machine, until they blew past an operating-system thread limit all at once. The full autopsy of that one waits in AWS Kinesis, 2020: The Thread-Limit Cascade; the lesson to carry now is that scale itself has sharp edges.

Suppose the worst has happened and the fleet is down. You fix the trigger, you bring the servers back, and you get hit by the cruelest twist in the whole subject: the recovery is its own cascade.

While you were down, every client kept queuing work and retrying. The instant your service comes back, all of that pent-up demand arrives at once — a thundering herd, the stampede from Stampedes, Thundering Herds & Cache Warming — and it lands on a service at its most fragile: caches cold and empty, connection pools unestablished, every request a cache miss that hits the backend. The freshly-restarted fleet, weaker than usual and hit harder than usual, falls over again before it can warm up. Restarting a crashed service into full traffic is how you turn one outage into three.

So recovery has to be deliberately gradual. Drop traffic hard first — allow only a trickle, one percent, so caches can warm and connections can form on a service that isn't also being crushed. Then ramp the load back up slowly, with jitter so the returning clients don't re-synchronize into a fresh herd, and lean on the graceful-degradation modes you built in Load Shedding & Graceful Degradation to serve cheaply while you climb back. Sometimes you even disable health checks briefly, so that tasks still starting up aren't killed for failing a check before they've had a chance to become healthy. And the warning that ties the section together: if you didn't fix the root cause — if the real problem was simply not enough capacity — the cascade will retrigger the moment full traffic returns, and you will be right back where you started. Don't let the cure become the next cause.

• A cascade is failure that spreads. A crack in one layer cracks the layer that calls it: a slow backend fills its callers' pools, whose exhaustion climbs upward until the user-facing root dies. A single local failure becomes a total outage.
• The death spiral feeds itself. Retries manufacture load (four retries across three layers is sixty-four attempts), and a dead server's traffic redistributes onto the survivors and tips them over too. Each death makes the next more likely.
• Hysteresis is the trap. A cascade sustains itself far below the load that started it — healthy at 10,000, cascading at 11,000, and not recovering until you drop to 1,000. Removing the trigger does not stop the loop.
• The whole section was defense in depth against this. Timeouts, retry budgets, breakers, bulkheads, backpressure, shedding, and tail-tolerance each cut a different link. No one of them is enough; a cascade climbs through the first gap you leave. Add capacity headroom, and remember scale itself has edges.
• Recovery is another cascade. A restarted service meets a thundering herd on cold caches and falls over again. Recover gradually: a trickle of traffic, warm up, ramp with jitter, degrade, and fix the root cause or it all comes back.

Here is the uncomfortable truth that makes this the second-to-last idea in the section. A cascade does not live inside any single component; it lives in the interactions between them — in how a timeout in one service meets a retry policy in another meets a pool size in a third. That means you cannot find it by reading the architecture diagram, however carefully. The links that will fail are emergent, invisible until they're pulled tight under real stress. The only honest way to know your system survives a cascade is to cause a small one on purpose, in production, under control, and watch what actually happens. Deliberately breaking your own system to find its cascades before they find you is the last discipline of this section: Chaos Engineering.