Checkpoint: Replication & Consistency

You started this section with a single database and a copy of it, and a vague sense that "replication" meant "a backup, but faster".

Since then you have watched a conflict get born out of nothing but two people saying yes at once. You have priced the two dials that decide how much data you lose and how long you are down. You have proved R + W > N with the pigeonhole principle, and then watched it fail to save a write anyway. You have read the actual CAP proof and discovered that the famous version of it is wrong. You have watched a perfectly healthy cluster get fifty times slower with nothing broken at all. You have seen a deleted user come back from the dead. You have watched a database destroy a customer's order because a machine's clock was five seconds fast. And you have run a failover with your own hands and paid the bill in five real, acknowledged, paid orders.

Nine lessons. Nine sets of knobs.

And every single one of them was the same knob.

Here is the thing that took me nine lessons to be able to say in one line.

A distributed system is not a pile of data. It is a pile of news that has not arrived yet.

Information moves at a finite speed. That is not an engineering problem you can fix with a better database; it is a fact about the universe. And it means that every decision in this section — the topology, the commit dial, the failure detector, N/R/W, the consistency level, the conflict-resolution strategy — is one question wearing nine costumes:

How much do you wait to hear, before you act?

• Wait less → you answer faster… and eventually you are wrong. Stale reads. Lost writes. Conflicts. Zombies.
• Wait more → you are correct… and eventually you are unavailable. Because "waiting", carried far enough, has another name.

There is no setting that is neither. That is not pessimism; it is the shape of the problem.

And you can never wait exactly enough, because of one impossibility that seeds all nine lessons — the one Gilbert and Lynch state in Corollary 1.1 of the CAP paper:

"…no way of determining whether a message has been lost, or has been arbitrarily delayed."

You cannot tell a slow node from a dead one.

Everything else in this section is a consequence of that sentence. Your timeout is not a configuration value; it is the moment you give up waiting and start guessing — and CAP's own §4.3 says so out loud: after the timeout, the node "concludes that the message was lost." It concludes. It guesses.

Read this table slowly. It is the section, restated in one sentence each — and the point is not the nine rows, it is that they are all the same row.

the lesson	…is really asking
Replication Topologies	Who is allowed to say yes? — i.e. how many nodes must have heard before we call it written. One, some, or any.
Sync vs Async & Failover	Two dials, and both are waits. How long do I wait to hear from the replica before I ack the customer (RPO)? How long do I wait to hear from the leader before I declare it dead (RTO)?
Quorums: N, R, W	Insist on hearing from enough nodes that at least one of them heard the write. That is all R + W > N says.
The Consistency Spectrum	Each rung = how much news you insist has arrived before you answer. Which is why only the linearizable rung pays a network round-trip: it is the only one that had to go and ask.
CAP, Properly	Answer without the news, or wait for news that may never come. The theorem, in one line.
PACELC	"A partition is just infinite latency." It was one dial the whole time — and your timeout is where "slow" gets legally reclassified as "down".
Read Repair & Hinted Handoff	The news arrives late, and somebody has to carry it. Eventual consistency is not a property your database has — it is a chore.
Version Conflicts	Two writes where neither heard the other. The absence of news IS the conflict — and a timestamp cannot tell "I saw you and chose otherwise" from "I never heard of you".
Codelab: Postgres Failover	The standby never heard about five paid orders. And split-brain is precisely a machine that never got the news that it was fired.

That last row is the one I would keep, if I could only keep one.

An old primary accepting writes after a failover is not a mystery, or a bug, or bad luck. It is a computer that has not yet received the news that it is no longer in charge — and no software running on that computer can deliver that news to it, because a machine that is paused or wedged cannot execute the instruction to stand down. Which is why the last line of defence in real high-availability systems is not software at all. It is a watchdog that resets the machine and does not ask nicely.

Enough theory. Here are three products, three sets of hard constraints, and the dials you have spent nine lessons learning.

A warning before you start: not one of the three answers is the one you will reach for first. One of them cannot be solved with these dials at all — and finding that out for yourself is the single most useful thing this checkpoint can do for you.

Each product opens on the setting a careful, nervous, well-intentioned engineer actually picks. Each of those settings fails. Go and find out why.

The shopping cart. Your instinct is safety: R+W > N, so W=2, R=2. But W=2 means a lonely replica on the minority side of a partition cannot accept a write — Unavailable: required_replicas: 2, alive_replicas: 1 — and you just refused the add-to-cart you swore you would never refuse. So drop to W=1, stay open… and now concurrent adds collide and last-write-wins destroys one of them. Try every combination of the dials. There isn't one that works. ⭐ The knob is the wrong knob: model the cart as a set and let it union-merge. Nine lessons of consistency levels, and the cart is rescued by a schema change.

While you are in there, try the half-fix — the one everybody reaches for when they see a stale read. Leave W=1 and raise the read to R=2. It feels like it should work. 1 + 2 = 3, which is not greater than 3. A quorum read alone buys you exactly nothing: there is still a read set that misses the one node that took the write. You paid the latency and bought no guarantee. You have to pay on the write too.

The ledger. "It's money, so: ALL." But R = N tolerates zero node deaths, and with synchronous replication a dead replica hangs the healthy leader's commits (8,101 ms, measured, and it would hang forever). The safest-sounding setting is the least safe one. QUORUM — W=2, R=2 — never lies and survives a death.

"Last seen 2 minutes ago." "QUORUM, to be safe — it was basically free in our rack." And it was: 8.43 ms vs 8.03 ms, same rack, measured. That benchmark is the trap. Move the replicas 70 ms apart and the same setting costs 73.20 ms instead of 1.45 ms — a 50× tax, on a perfectly healthy cluster, to protect a value whose staleness literally nobody can perceive. ⭐ W=1, R=1 is not a compromise here. It is correct: last_seen_at is a snapshot, and last-write-wins is exactly right for a snapshot.

⭐⭐ And now look at what you just did. The cart needed a set. "Last seen" needed last-write-wins. Same knob — opposite answers. The thing that decided it was not a consistency level. It was asking what the value is: an accumulation, or a snapshot?

Eight calls. Every wrong option is a trap this section warned you about by name — they are lifted straight from the nine Mental-Model Corrections. If one of them looks obviously right to you, that is worth knowing.

Pass mark: 6 of 8.

Concretely — things you could not do nine lessons ago:

• Read R + W > N as arithmetic, not folklore. And say precisely what it buys (the latest completed write, absent concurrent writes) and what it does not buy (linearizability).
• Refuse to answer "is our system CP or AP?" — and explain that CAP is a property of a request, and that your own service is probably making different choices on different endpoints right now.
• Price consistency in milliseconds, on a healthy cluster, using nothing but the distance between your replicas.
• Spot the half-fix (W=ONE + R=QUORUM) in a design review, and do the arithmetic out loud.
• Look at a value and ask the only question that matters for conflicts: is it a snapshot (LWW is correct, and free) or an accumulation (LWW is silent data loss, and it's the default)?
• Run a failover, and know that the dangerous steps are the ones either side of promote — fencing before, rewinding after.
• Argue for wal_log_hints = on on a quiet Tuesday, because you know what it costs to not have it at 3am.
• Say "a replica is not high availability" — and be able to list exactly what else is required.

• ⭐⭐⭐ The law: a distributed system is a pile of news that has not arrived yet. Every knob in this section is one knob — how much do you wait to hear, before you act? Wait less → wrong. Wait more → down. There is no third option.
• ⭐⭐⭐ You cannot tell a slow node from a dead one (G&L Corollary 1.1). Your timeout is not a config value — it is the moment you stop waiting and start guessing.
• ⭐ The dials are not the answer; they are the question. A shopping cart cannot be fixed by any N/R/W setting. It is fixed by a set.
• ⭐ "ALL" is the least safe setting. It tolerates zero deaths, and a dead replica hangs a healthy leader (8,101 ms, measured). QUORUM is the safe one.
• ⭐ The consistency tax is paid in distance, not in failures. 8.03 ms in one rack; 73.20 ms with replicas 70 ms apart — nothing broken, no partition, 50× slower.
• Snapshot or accumulation? That one question decides whether last-write-wins is correct and free or silent data loss — and it is the default either way.
• Split-brain is a machine that never got the news that it was fired — which is why you cannot solve it on the node you are trying to stop, and why the last line of defence is a watchdog.

Next: §4 — Scaling Reads. You now know exactly what a replica costs you in truth. Time to find out what it buys you in throughput — and to meet the staleness you just learned to price, wearing a new hat.