Checkpoint: Learning from Real Outages

For six lessons you learned reliability as a set of clean laws — how to stay up. Then this section did something different: it took you inside five real disasters at companies that knew those laws cold and went down anyway. A company that erased itself from the internet with one command. A fleet that melted because someone added servers. A region so much of the internet leans on that its bad morning is everyone's. The generic anatomy hiding inside all of them. And the skill of reading any of it for the real lesson.

This checkpoint is where those five stories stop being war stories and become judgment. First we'll compress the whole section into a single law and a single reflex — the when you see this shape, reach for that move that a seasoned engineer runs without thinking. Then you'll take the exam: ten real situations, each with a tempting wrong turn these outages taught you to refuse. Get seven of ten and you've earned the judgment. Miss a few and you'll know exactly which outage to revisit before you move on.

The whole section compresses to one sentence per outage. Say these five and mean them, and you carry the section in your head:

• The tools you fix an outage with cannot ride the thing that's down. A recovery path that shares fate with the failure is a circular dependency; you need an out-of-band, break-glass path, from Facebook, 2021: The BGP Self-Lockout.
• You cannot scale out of a cost that grows faster than your load. When each node's work grows with the whole fleet, adding capacity crosses a hidden ceiling; make the cost linear with bigger boxes, cells, or gossip, from AWS Kinesis, 2020: The Thread-Limit Cascade.
• Redundancy that shares a failure domain is not redundancy. Concentration turns independent-looking copies into one correlated bet, and a global control plane can trap your failover in the very region you're fleeing, from us-east-1: The Region the Internet Leans On.
• A cascade is a state you get stuck in, not an event that passes. The trigger is interchangeable; the sustaining loop is the root, and once you're metastable you break the loop rather than out-run it, from Anatomy of a Cascading Failure.
• A postmortem is evidence to interrogate, not a verdict to absorb. Read it in foresight, distrust the single root cause, spot the blame and the vague fixes, and always ask does my system have this shape, from How to Read a Postmortem.

Read together they say one thing: real outages are not freak events; they have a shape, and the shape is learnable.

Laws are worth little until they're a reflex — the move your hand reaches for before you've finished reading the page. Here's the section as a decision flow: the situation on the left, the move it should trigger on the right.

• "I know what's broken, but I can't reach the tools to fix it." You have a circular dependency. Stop trying the normal path and go out-of-band — the break-glass path that shares nothing with the failure.
• "It's overloaded, so I'll add servers." First check whether each node's cost grows with the fleet. If it does, adding servers makes it worse; go bigger and fewer, or split into cells, or switch to gossip.
• "We're multi-region, so a regional outage can't take us down." Check where your auth and your failover controls live. If they route through us-east-1's control plane, you're not as independent as you think; use regional endpoints and design for static stability.
• "It's still down, but the thing that started it is long gone." It's metastable. Do not add capacity and do not just wait — break the sustaining loop by shedding load and capping retries, then ramp back slowly.
• "I'm reading someone's postmortem." Read the timeline in foresight, distrust the single root cause and hunt the contributing factors, flag the blame and the vague action items, and finish by asking whether your own system has the same shape.

That's the reflex. The exam below is nothing more than these situations, dressed up as stories with a plausible wrong answer in each — the trap you now know how to walk past.

Ten decisions, each a real situation with a real trap. For each one, pick the move you would actually make in production, not the textbook phrase — the decision. You'll get an immediate verdict, the one-line principle behind it, and the exact outage to revisit if you miss it. Seven out of ten is a pass. Take your time; a wrong answer here is the cheapest kind of lesson.

Step back and notice what you can do now that you couldn't before this section. Handed a real outage, or your own system, you can:

• Spot a circular dependency in a recovery plan, and insist on an out-of-band, break-glass path that shares nothing with the thing it recovers.
• Recognize a super-linear cost hiding in an all-to-all design, predict the hidden ceiling it will hit, and restructure it to grow linearly before it bites.
• Audit your real blast radius, including the vendors and global control planes you didn't know you depended on, and decide on purpose which concentration risk you accept.
• Read a cascade by its stages — vulnerable state, trigger, sustaining loop, metastable collapse — and know that once it's stuck, you break the loop instead of adding capacity.
• Read any postmortem on the internet like a professional, in foresight, skeptical of the single root cause, alert to the red flags, and always converting it into your own action items.

That is the judgment of an engineer who has, in effect, survived outages they never had to live through, by learning to see the shape.

• Five different triggers, one animal underneath. A command, a capacity add, a DNS record, a config push, a badly written report — the specifics vary, the shape repeats.
• Every trap in the exam was the comfortable option. Retry the dead dashboard, add servers to a melting fleet, trust that multi-region means safe, wait for a metastable failure to clear, believe a postmortem's tidy single cause. The wrong move almost always feels reasonable.
• Learning from failure is a skill, not luck. You can install the scar tissue of a far more experienced engineer by reading other people's disasters and asking, every time, would I have caught this, and does my system have it?

You've learned the reliability discipline as laws, and then watched five famous systems break those laws in public. There's one thing left: to put it all together on a system of your own. The capstone hands you the marketplace you've been building all course and throws real failures at it — a lost region, a retry storm, a flash crowd — and asks you to keep it standing, in Capstone Pass 3: The Marketplace Survives.