Checkpoint: The Reliability Discipline

Eight lessons ago, reliability was a vague hope — we'll try to keep it up. Now it's a discipline, a connected set of moves for a system that is always a little broken and must stay useful anyway. You learned to define and measure what reliable even means, to wake the right person and no one else, to run an incident calmly and learn from it, to recover from a lost region or a corrupted database, to shrink a failure to a single cell, to keep your data both durable and correct, and to ship changes without betting the whole system on them.

This checkpoint is where those eight lessons stop being separate ideas and become one reflex. First we'll compress the whole section into a single law and a single decision flow — the when this, reach for that that a strong engineer runs without thinking. Then you'll take the exam: ten real situations, each with a tempting wrong turn that these lessons taught you to refuse. Get seven of ten and you've earned the discipline. Miss a few and you'll know exactly which lesson to revisit before you move on.

The entire section compresses to one sentence per lesson. If you can say these eight lines and mean them, you have the spine of the whole thing:

• You can't defend what you can't measure. Turn reliability into arithmetic with an SLI, an SLO, and an error budget that is your permission to ship , from SLIs, SLOs & Error Budgets.
• Wake a human only for a symptom they must act on now, and only as fast as the budget is burning; everything else is a dashboard , from Alerting Philosophy.
• When it breaks, one person coordinates and stops the bleeding before anyone hunts the cause, and afterward you blame the system, not the human , from Incident Response & Postmortems.
• Some failures you can't roll back, so plan recovery in advance, measured by how much time (RTO) and data (RPO) you can afford to lose , from Disaster Recovery: RTO & RPO.
• Running everywhere buys latency and survival but taxes every write — across regions you get low latency or strong consistency, never both , from Multi-Region Patterns.
• Stop building one system that fails completely; build many small cells that fail one at a time, and keep the router in front of them dumb , from Cell-Based Architecture & Deployment Stamps.
• Durable is not correct. A checksum catches a flipped bit, but only a backup from before catches a valid wrong write — and only if you've restored it , from Data Integrity: Backups That Restore.
• Test the change against production's own traffic before any user depends on it — mirror it in the shadows, or ship it dark behind a flag , from Shadow Traffic & Dark Launches.

Read together, they say one thing: reliability is not the absence of failure; it's the discipline of staying up anyway.

Principles are worth little until they're a reflex — the move your hand reaches for before you've finished reading the alert. Here's the section as a decision flow: the situation on the left, the move it should trigger on the right.

• "How reliable is this, really?" Measure it. Define an SLI, set an SLO, and track the error budget — stop arguing and start counting.
• "Should I wake someone?" Only if a user-facing symptom is burning the budget fast. Otherwise it's a ticket or a dashboard, not a page.
• "It's on fire right now." Mitigate first — roll back, fail over, flip a flag — with one person commanding. Diagnose later, blamelessly.
• "The region is down / the data is gone." Recover to your RTO and RPO. If the data is corrupt rather than lost, restore a clean backup — replication won't save you.
• "My users are far away, or I must survive a whole region." Go multi-region, but pick a write strategy and accept the tax: low latency or strong consistency, not both.
• "One bad request or deploy took everything down." Shrink the blast radius: cells, shuffle sharding, a dumb router — so the next one takes down a slice, not the world.
• "Did my data rot, or did something mangle it?" Checksums and scrubbing for rot; a restorable, point-in-time backup for a bad write. Know which problem you have.
• "Is this new version actually safe to ship?" Prove it in production first — shadow it (read-only!) or dark-launch it behind a flag, then ramp with the kill switch ready.

That's the reflex. The exam below is nothing more than these situations, dressed up as stories with a plausible wrong answer in each — the trap you now know how to walk past.

Ten decisions, each a real situation with a real trap. For each one, pick the move you would actually make in production — not the textbook phrase, the decision. You'll get an immediate verdict, the one-line principle behind it, and the exact lesson to revisit if you miss it. Seven out of ten is a pass. Take your time; a wrong answer here is a free lesson, which is the cheapest kind.

Step back and notice how much you can now do that you couldn't eight lessons ago. Given a real production system, you can:

• Define and measure reliability with SLIs and SLOs, and spend an error budget deliberately instead of chasing a perfect number.
• Design an alerting system that wakes the right person for the right reason and stays silent for everything else.
• Run an incident with a clear commander, mitigate before diagnosing, and write a blameless postmortem that fixes the system so the same fire can't start twice.
• Build a disaster-recovery plan with honest RTO and RPO targets, the right recovery tier for the budget, and the discipline to actually drill a restore.
• Architect across regions, choosing active-passive or active-active and a write strategy with eyes open to the consistency-versus-latency tax and the split-brain risk.
• Shrink the blast radius with cell-based architecture and shuffle sharding, so no single failure can take down more than a slice.
• Keep data both durable and correct, defending silent rot with checksums and logical corruption with backups you have actually restored.
• Ship changes safely by testing them against production's real traffic with shadow deployments and dark launches, ramping behind a flag with a kill switch.

That is the toolkit of an engineer who can be handed a system and trusted to keep it up.

• The discipline is one reflex, not eight topics. Measure it, alert on it, respond to it, recover it, spread it, isolate it, protect its data, and ship to it safely — each move for its moment.
• Every hard case in the exam was a valid-looking wrong turn. The trap is almost always the comfortable option: keep freezing a healthy budget, page on the cause, diagnose before mitigating, trust a replica to fix corruption, put logic in the router, or forget that a discarded shadow response still charged the card.
• Reliability is a posture, not a project. It's not a milestone you finish; it's the set of habits that let a system fail small, recover fast, and stay honest about how well it's really doing.

You've learned the discipline as principles — clean laws about how to stay up. The fastest way to make them permanent is to watch them get broken. The next section takes you inside real, famous outages at companies that knew all of this and went down anyway, and asks the only question that matters: which of these laws did they miss, and would you have caught it? It opens with the day a single change locked an entire company out of its own network, in Facebook, 2021: The BGP Self-Lockout.