Checkpoint: Consensus & Coordination

The codelab ended with a bill: ten decisions, drawn from everything between FLP and the herd effect. This is where it's paid.

Nine lessons ago, "the cluster agrees" was a phrase you had to take on faith. Since then: agreement got a contract and a price tag, majorities got a proof, elections got terms, the log got its three clauses, the source got its courier, news learned to travel free, locks learned to survive their owners, the recipes moved into a shop you can buy from, and all of it ran live in your terminal. The exam below never asks for a definition. It hands you a room with something on the line and asks what you do.

One compression before the exam. Every lesson here is the same law in a different costume:

A decision is one answer the system never takes back. Buying one costs overlap, a numbered round, and an owner — so buy decisions only where two answers would ruin you, and let news travel free.

• The Consensus Problem: the contract and the price. Agreement, validity, termination; FLP's deal is safety always, liveness usually, because dead is indistinguishable from slow.
• Quorum Intuition, Revisited: the venue. Two majorities must share a member — pigeonhole, not luck. But overlap is the venue, not the verdict.
• Leader Election: the owner. Symmetric rules make concurrent candidacies safe, random timers make them rare, and the term is authority as a number that only goes up.
• Raft, Visually: the machine. A replicated log plus three clauses; agreement on state is agreement on order.
• Paxos, the Intuition: the source. The courier rule — later ballots don't threaten a chosen value; they deliver it.
• Gossip Protocols: the counterpoint. Spreads, never decides. Rumors for speed, sweeps for the promise.
• Distributed Locks & Leases: the boundary. A lease plus a pause equals two holders; fencing doesn't prevent the overlap, it defuses it.
• ZooKeeper & etcd in Practice: the shop. Coordination is a service you buy, not a protocol you write; the tokens are already minted.
• Codelab: Leader Election with etcd: the receipt. Blocking is holding, and the abstraction and its implementation told the same story.

Read the law once more and notice its two halves. The first half is the machinery: overlap, rounds, an owner. The second half is the judgment: most of what a cluster shares is news, and news doesn't need the machinery at all.

Four of the exam's questions hand you a need and make you shop, because that's the form this knowledge takes in real design reviews. The whole method is three forks and a standing rule:

Fork one: news or a decision? If two nodes briefly disagreeing ruins nothing — who's alive, what build is where, metrics — it's news. Gossip carries it for a constant few messages per node, survives anything, and never decides.

Fork two: if it's a decision, does a double-run waste work or ruin data? Waste means the efficiency lock: one key, SET NX, an expiry, done — correctly cheap. Ruin means the full purchase: a consensus-backed lock service, a lease, and a fencing token checked at the resource itself, because a lease plus a pause equals two holders.

Fork three: one decision, or a history of them? A history every replica must replay identically is the replicated log — bought as a service (etcd, ZooKeeper, or the consensus inside your database), never written by hand. And when the shared thing is small, load-bearing config that must push to dozens of readers, that's the notice board: watched keys, re-armed, strongly consistent.

The standing rule: every rung south of gossip pays the same three taxes — majorities, round trips, and the pause. If a proposal claims to skip one of them, you've found the part that breaks at 3 a.m.

Ten scenarios from the rooms where this section earns its keep: purchase decisions, a vendor pitch, a capacity review, an incident channel, a code review of a Paxos proposer, and one bug report that isn't a bug. Every wrong option is a real wrong turn these nine lessons taught you to refuse. Pass is seven of ten; miss one and the verdict names the exact lesson to revisit.

If this section landed, here is the capability it certifies.

In a design review, you can hear "the standby promotes itself when the primary stops answering pings" and name it: a hand-rolled consensus protocol, usually a doomed one. You can size an ensemble and defend the odd number with arithmetic. You can ask the fork's question — waste or ruin? — before anyone says the word lock, and refuse a bare lease for a ledger with the same confidence you'd accept one for a cache warmer.

In an incident, you can read a stalled succession and find the unrevoked lease behind it. You can watch a woken zombie leader abdicate and know which number made it kneel. You can spot a thundering herd at the coordination layer and fix it with one arrow: watch your predecessor.

And under all of it, you know what the machinery costs and when not to buy it: consensus for the decisions, gossip for the news, and a fencing token wherever a decision touches something that can't defend itself.

That vocabulary — the contract, the pigeonhole, the term, the log, the courier, the wave, the lease, the recipes, the receipt — is assumed from here on.

• The law: a decision is one answer never taken back; it costs overlap, a numbered round, and an owner. News travels free. Nine lessons, one sentence.
• The nine costumes, one line each: the contract and FLP's deal · majorities can't split, but overlap alone can't decide · the term is authority that only rises · the log makes order durable · the courier delivers chosen values · gossip spreads and never decides · a lease plus a pause equals two holders, so fence the resource · coordination is a service you buy · and the terminal showed the abstraction and the implementation agreeing.
• The ten-second purchase: news or decision, waste or ruin, one or a history. Then pay the three taxes with open eyes: majorities, round trips, the pause.
• Passing means permission. Seven of ten says you can shop for coordination under pressure, and everything ahead assumes exactly that.

Now look at the door this whole section guarded. Every tool here funnels a decision through one place: a majority, an owner, one answer, never taken back. The next section opens with the systems that refuse the door on purpose — two people editing the same document on two continents, a phone that syncs after a week offline, regions that keep accepting writes while the cable between them is cut. They accept two answers, on purpose, and then have to face what they've done. The clocks lessons gave you the tool that proves two writes are concurrent; what it never said is what to do with the pair. That reckoning is next: The Conflict Problem.