Checkpoint: Why Distribution Is Hard

Six lessons ago, this stretch of the course made you a promise: before you're asked to build anything that survives, you'll get an honest diagnosis of what you're up against. That diagnosis is now complete. The network lies, in eight specific ways. Nodes fail in shapes, from a polite halt to a confident lie. A cut cable turns one cluster into two, each convinced it's in charge. Detectors don't detect — they bet. And the most expensive failures never show up on the dashboard at all.

But a map is only yours once you can navigate with it while something is on fire. That's what this checkpoint is for: ten scenarios lifted from the kinds of design reviews and incident calls where these ideas earn their keep. No definitions, no trivia: every question is a decision, and every wrong option is a wrong turn the last six lessons taught you to refuse.

Before the exam, one compression. Strip the six lessons to the bone and the same sentence is written on every one of them:

A distributed system acts on evidence, never on truth — and the evidence lies.

No node sees the whole system. Each machine reasons from whatever makes it across the wire: replies, heartbeats, probes, and the silences between them. That's evidence, not truth, and the entire section is that one gap wearing six costumes.

• Welcome to Hard Mode named the gap. Dead is indistinguishable from slow, because a dead node, a slow node, and a cut cable hand you the exact same evidence: silence. A timeout converts that silence into a guess, never into a fact.
• The Fallacies of Distributed Computing listed the eight ways you'll trust the evidence channel more than it deserves, starting with the mother lie: the network is reliable.
• Failure Models: Crash, Omission, Timing ranked the ways evidence goes bad: by how honestly a node tells you it's broken, crash to Byzantine, each rung with a price tag.
• Network Partitions & Split Brain showed machinery acting on wrong evidence: your own failover crowns a second king, and forty-three seconds of cut buys a day of divergence.
• Heartbeats & Failure Detection priced the guessing itself: the detector is k × T, a bet with a loser on each side.
• Gray Failures is the endgame: evidence and reality diverge on the same machine. The dashboard swears green while users bleed, and the redundancy you paid for waits politely for a detection that never comes.

Six topics on the syllabus; one enemy in the field. The exam below never asks you to recite a definition. It asks whether you recognize the costume when it walks into your incident channel.

There's a practical habit hiding inside those six lessons, and it deserves to be explicit before you're tested on it. This is what the section wants your hands to do in the first sixty seconds after a call doesn't come back.

Move one: distrust your own eyes. Your view is one witness, and the judge itself may be sick. Ask peers to probe the suspect before you convict. If their calls sail through, the broken thing is your route, not the node. And compare the two views: what do the suspect's users see? Probes green while users bleed is not a contradiction; it's a diagnosis.

Move two: name the rung. Halted and honest about it — crash. Sometimes silent, otherwise correct — omission. Right answers, wrong time — timing. Confidently wrong — Byzantine, and before you camp on that rung, remember that checksums demote most of it to garbage you can simply drop. The rung matters because the price does: restart, retry, timeout, and vote cost wildly different amounts, and paying the top-rung price for a bottom-rung fault is how teams drown in their own machinery.

Move three: act at the price of your certainty. Reversible moves are cheap on mere suspicion: route traffic away, shed some load, start warming a spare. Irreversible moves wait for agreement: promotion needs a quorum's verdict, and anything that writes needs the fence, the token check at the resource itself, armed for the day the verdict is wrong anyway.

Confirm collectively, classify honestly, act reversibly. The figure below is the version to keep taped above your desk.

Ten scenarios. Each is a decision someone had to make with incomplete evidence, which is to say, each is a normal Tuesday in a distributed system. Pass is seven of ten. Miss one and the explanation tells you why on the spot; miss a few and the final verdict names the exact lessons to revisit. Nothing below tests anything the section didn't teach.

A checkpoint certifies capability, not coverage. If this section landed, here is what you can do that you couldn't six lessons ago.

In a design review, you can read a sentence like “the services call each other over the internal network, so we treat those calls as local” and name the exact assumptions being smuggled past you, then ask what happens when each one fails, in which shape, at which rung. When someone says “we handle failures”, you can ask the question that separates a slogan from a design: which class?

In an incident, you can hold a timeout's verdict at arm's length (a guess, not a fact), probe from a second vantage point before convicting, and match the response to your certainty: reversible on suspicion, irreversible only on agreement.

And when the dashboard and the support queue disagree, you no longer call it a paradox. You know the two witnesses walk different roads, and you know which road the truth about your users lives on.

That vocabulary (silence, rungs, quorum, the fence, k × T, suspicion, the two views) is the working language of everything ahead.

• The law of the section: a distributed system acts on evidence, never on truth — and the evidence lies. Silence is ambiguous, probes walk different roads than users, and every detector verdict is a bet.
• The six disguises, one line each: a network of machines lies (the territory) · eight named assumptions will betray you (your own head) · failures come in rungs with prices, crash → omission → timing → Byzantine (the shapes) · acting on a wrong guess forks history: quorum for the tiebreak, fencing for the zombie (the damage) · detection is k × T, a dial with a loser on each side (the bet) · gray failures pass every check while users bleed, so measure from the application's side (the blind spot).
• The triage habit: confirm collectively, classify honestly, act reversibly. Suspicion is cheap; conviction is expensive; the fence is for the day you're wrong anyway.
• Passing means permission. Seven of ten says you can use the map, not just recognize its landmarks — and everything ahead assumes exactly that.

One lie is still standing, though. All section long you've read logs and told stories like “the standby promoted after the primary froze” as if every machine agreed on what after means. They don't. Every machine keeps its own clock, every clock drifts, and “before” is about to stop being a simple word. That's where the course goes next: The Clock-Sync Problem.