Anatomy of a Cascading Failure

Three outages, three completely different triggers. A maintenance command that erased Facebook from the internet. A routine capacity addition that melted a fleet at Amazon. A bad DNS record that took a chunk of the web offline. If you were only watching the triggers, you'd file these as three unrelated accidents and move on.

But you weren't only watching the triggers, and by now something should be nagging at you: underneath, they were the same animal. A small event. An amplifier that turned it into a big one. A system that got stuck being broken, well after the original cause was gone. A recovery that was strangely slow and couldn't just be brute-forced. Peel off the specifics of each story and the same skeleton is standing there in all three.

This lesson dissects that skeleton, so you never have to be surprised by the fourth one. We're going to name the stages every cascade moves through, and then, using the modern research on how these failures actually behave, we'll answer the two questions those three outages kept raising and never quite explained: why couldn't they just undo the change? and why couldn't they just add more servers? The answer to both is a single idea, and once you have it, you'll read every outage report differently.

In this lesson: the anatomy of a cascade, stage by stage; why a cascade is a state you get stuck in, not an event that passes; and the one skill that separates people who stop these from people who make them worse, knowing where in the anatomy you are and where to cut.

Scope: the mechanics of how failure climbs a call graph, and the specific defenses that cut each link, are built in Cascading Failures. This lesson is the layer above it, the diagnostic shape and the timing.

Lay the three case studies side by side against four stages, and the pattern jumps out.

The trigger is the spark. For Facebook, 2021: The BGP Self-Lockout it was one maintenance command; for AWS Kinesis, 2020: The Thread-Limit Cascade it was a small capacity addition; for us-east-1: The Region the Internet Leans On it was a DNS record gone bad. Three different sparks, and here's the first surprise: the spark is the least interesting part.

The amplifier is what turned a spark into a fire. Facebook's health check withdrew every route at once; Kinesis's thread count grew with the square of the fleet and crossed a hidden ceiling; us-east-1's shared dependency meant one service's failure was everyone's. The sustaining loop is what kept the fire burning after the spark was gone: Facebook's recovery tools ran on the dead network, Kinesis's servers were all left holding useless shard-maps, us-east-1's failover path depended on the region that was down. And the recovery was the same shape every time, slow, careful, and impossible to rush, a physical walk-in for Facebook, a throttled restart for Kinesis, hours of patience for us-east-1.

Same skeleton, three costumes. Once you can see it, an outage stops being a wall of scary specifics and becomes a diagnosis: which stage am I in, and what does this stage need? The rest of the lesson walks the skeleton one bone at a time.

Every cascade starts long before the outage, in a state that looks perfectly healthy. This is the part people miss, because there's nothing to see.

A busy system is almost always tuned to run close to its limits. That's not sloppiness, it's economics: capacity costs money, so you provision enough to handle normal load with some headroom, and not a lot more. Every optimization you add for the common case, a cache that usually hits, a fast path that usually applies, quietly pushes you to run at a higher multiple of what the system could survive if that optimization stopped working. You end up humming along a hair below a cliff you can't see, for months, with green dashboards the whole time.

The researchers who named this behavior call it the vulnerable state: stable, but only just, sitting near the threshold between fine and not-fine. Nothing is wrong. Nothing needs fixing. The system could run this way for years. But the closer you're tuned to the edge, the smaller the shove it takes to go over, and the whole point of stage one is that the shove has not come yet. The crack is already in the wall; it's just holding.

Then something nudges the system over the edge. A deploy. A traffic spike. A dependency that slows down or disappears. A config command. Google's site-reliability engineers put it bluntly: the most common trigger of all is simply overload, load arriving faster than a service was built to serve.

Here's the counterintuitive claim this lesson will keep making: the trigger barely matters. It feels like the cause, because it's the thing that changed and the thing your timeline starts with. But triggers are interchangeable. The same system, in the same vulnerable state, could have been tipped over by any of a dozen different sparks, and the resulting fire would look identical. Facebook could have lost its backbone to a bad command or a fiber cut; the cascade downstream would have been the same.

This matters enormously for how you respond and how you prevent. If you treat the trigger as the root cause, you'll fix that one trigger, write don't run that command again in the postmortem, and feel safe, right up until the next, different spark finds the same vulnerable system and knocks it over again. The trigger is where the story starts. It is not where the story lives.

If the trigger isn't the root cause, what is? This is the heart of the lesson, and it's the idea that makes cascades finally make sense: the real cause is a sustaining loop, a positive feedback loop that, once it starts turning, keeps itself turning.

The most common one, by far, is retries. Walk the circle. The system is overloaded, so some requests fail. The clients that made those requests don't give up, they retry. But a retry is another request, so now there's more load than before, which causes more failures, which trigger more retries. Round and round: overload feeds failure feeds retries feed overload. The loop is a machine for converting a small, brief problem into a large, self-sustaining one, and you built it on purpose, because retries usually make you more reliable, smoothing over the occasional blip. The same mechanism that saves you on a normal day is the one that damns you on a bad one. Retries aren't the only sustaining loop, a cold cache after a restart floods the backend the same way, and load shed from a dead replica onto its neighbors does too, but retries are the one you'll meet most.

Sit with the shift in perspective. The trigger is a spark you can't fully prevent, there will always be sparks. The loop is the pile of dry wood you chose to stack next to it. And the loop, not the spark, is what you have to break to put the fire out, which is exactly why the next stage is so cruel.

Here is the payoff, the single idea that explains every why couldn't they just... question the last three lessons raised. A system with a sustaining loop has two stable states. One is the healthy state you know: high throughput, requests served, everyone happy. The other is a degraded state that is also stable, throughput on the floor, the loop spinning, and, critically, it stays there on its own. Engineers call this second state metastable, and the system falling into it is a metastable failure.

The defining, horrible property is this: the failure persists even after the trigger is gone. The traffic spike passed ten minutes ago. The bad command was reverted. The slow dependency recovered. And your system is still down, because the retries it generated are now the load, and that load keeps causing the failures that keep causing the retries. The researchers describe it perfectly: the system is up, but down. Working, but broken. It's processing requests as fast as it can, and every single answer arrives after the client already gave up and left.

Now the two mysteries answer themselves. Why couldn't Kinesis just add capacity back? Because you cannot scale out of a metastable failure. Add fresh servers into a running retry storm and they're swamped the instant they come up, failing before they serve a single healthy request, so the loop just eats them. Why couldn't they just undo the change? Because the change, the trigger, was never what was holding the system down; the loop was. Removing a trigger that's already gone changes nothing. The only thing that ends a metastable failure is breaking the loop itself, and the console below lets you feel exactly that. Push a healthy system over the edge, watch it get stuck, and then try to save it, first with the obvious levers, then with the one that works.

Everything the anatomy teaches comes down to one practical skill: knowing which stage you're in, because the same fix that saves you early is useless late, and the fix that saves you late is overkill early.

Early, while the system is only tipping and the loop hasn't taken hold, almost anything works, and cheaply. Add a little capacity, undo the trigger, tighten a timeout, and the system settles back. This is the whole argument for catching cascades fast: a defense that would be a heroic emergency at minute twenty is a shrug at minute one. It's also the argument for the automatic defenses from Cascading Failures, a circuit breaker or a bulkhead or a load shedder acts in the first seconds, before a human has even read the page.

Late, once you're metastable, the toolbox shrinks to exactly one category: things that break the loop. The biggest hammer, straight from Google's playbook, is to drop load aggressively, shed a huge fraction of incoming traffic, retries included, until the system falls back under its capacity and the loop starves. It feels insane to throw away most of your traffic during an outage, but a service that crashes at 11,000 requests a second may need load cut all the way to 1,000 before it stabilizes, because so few servers are healthy enough to serve. Alongside it: cap the retries with a budget and jitter, from Retries, Budgets & Retry Storms, so the loop can't refuel; drain the backed-up queues of stale work nobody's waiting for anymore; and only then ramp traffic back slowly, because slamming it all on at once just starts the whole thing over, the thundering herd you met in the recovery of every case study.

The map is simple to say and hard to do under pressure: prevent by staying off the threshold and testing to failure in Chaos Engineering; intervene early with automatic loop-cutters; and if you're already stuck, break the loop before you try to out-run it.

You now have a lens, so let's make it a habit. The next time you read an outage report, or worse, live one, run the anatomy over it and ask four questions in order.

What was the vulnerable state? How close to the edge was this system running before anything went wrong, and why. That's your real exposure, and it was there for months.

What was the trigger, and what else could have been? Name it, then immediately ask what other sparks would have lit the same fire. If the answer is lots of things, you know not to waste the fix on the one trigger.

What was the sustaining loop? This is the one that matters, and the one incident writeups most often bury. Find the feedback, the retries, the redistribution, the cold cache, that kept the system down after the trigger passed. Fix this, and the whole class of outage goes away, not just today's flavor.

How did they break the loop, and how long did it take? The recovery tells you whether they understood the loop or just got lucky when the trigger happened to stay gone. If the fix was drop load and ramp slowly, someone knew exactly what they were looking at. If it was we kept adding capacity and it eventually held, they may have simply out-waited it. Reading outages this way is a skill you build deliberately, which is precisely what the next lesson, How to Read a Postmortem, is about.

Reason through this before you read the answer. A payments service runs happily at about 8,000 requests a second, sized for a peak of 10,000. One evening a popular sale sends a burst to 13,000. Requests start timing out; the mobile app, trying to be helpful, retries each failed payment up to three times. Ten minutes later the sale's traffic is back down to a normal 8,000, but the service is still at 100% errors and won't recover. An engineer doubles the server count. Nothing improves. What stage is this, and what actually fixes it?

Work it out, then check yourself:

• The stage: metastable. The trigger, the sale burst, is already gone (traffic is back to 8,000, well under the 10,000 capacity), yet the system stays down. That's the signature: the failure outlived its cause.
• Why doubling servers did nothing: you can't scale out of a metastable failure. The real load isn't the 8,000 real requests, it's those plus a storm of retries, and the fresh servers are swamped by the storm the moment they start. The loop eats the new capacity.
• What actually fixes it: break the loop. Shed load hard, reject most incoming traffic including retries so the service drops under capacity and the retry storm starves; cap the app's retries with a budget and backoff so it can't refuel the loop; then ramp real traffic back up slowly. The moment the loop is starved, the original capacity is plenty, because the real demand was only 8,000 all along.
• The tell to remember: when a system stays broken after its trigger is gone, and adding capacity does nothing, stop adding capacity. You're not short on servers. You're stuck in a loop, and the only exit is to break it.

The anatomy corrects a set of instincts that feel right and make cascades worse.

• Myth: the trigger is the root cause. The trigger starts the cascade; the sustaining loop is what makes it a disaster and keeps it alive. Triggers are interchangeable, so fixing one leaves you exposed to the next. Find and fix the loop.
• Myth: just add capacity. You cannot scale out of a metastable failure. Fresh servers dropped into a running retry storm are swamped before they serve anything. Break the loop first; then your existing capacity is usually plenty.
• Myth: once the trigger's gone, it'll recover on its own. A metastable state is stable, it sustains itself. The spike passed, the command was reverted, and the system is still on the floor, because the retries it spawned are now the load.
• Myth: retries make us more reliable. In the common case, yes, which is why they're everywhere. But they are the number-one sustaining loop, quietly turning a blip into a self-feeding storm. Retries are only safe with a budget, a cap, and randomized backoff.
• Myth: it was a rare, unpredictable freak event. A cascade is the default behavior of a system with no loop-breakers, tuned close to its edge. The vulnerable state was there the whole time; the trigger just happened to arrive.

• Every cascade shares one skeleton: a vulnerable state, a trigger, a sustaining loop, a metastable collapse, a blast radius, and a slow recovery. Learn the stages and any outage becomes a diagnosis instead of a surprise.
• The trigger is the least important part. It's interchangeable, and treating it as the root cause just protects you from that one spark. The sustaining loop is the real cause.
• A cascade is a state, not an event. With a feedback loop, a system has two stable states, healthy and metastable, and it can get stuck in the bad one, staying down long after the trigger is gone. Up, but down.
• You can't scale out of a metastable failure, and you can't undo your way out either. Fresh capacity gets swamped by the loop; the trigger you'd undo is already gone. Only breaking the loop works.
• Timing is the whole skill. Early, almost any fix works and works cheaply. Late, once you're metastable, drop load hard, cap retries, drain queues, then ramp back slowly, and only then.
• Read every outage with the anatomy in hand: what was the vulnerable state, what was the trigger and what else could have been, what was the loop, and how did they break it.

That last habit, reading a failure for its skeleton instead of its headline, is a craft of its own, and it's the one that turns other people's disasters into your own hard-won experience. The next lesson makes it explicit: How to Read a Postmortem.