Replication Topologies

At 14:02:31, a customer in Frankfurt updates her shipping address. At 14:02:31, a support agent in Virginia — looking at the same account, trying to help — updates it too. Two writes, same row, same second, two continents.

In one architecture this is a complete non-event. The database picks an order, one write lands after the other, and nobody ever knows it was close. In another architecture, both writes succeed, both machines are certain they're right, and nobody finds out there's a problem until minutes later — when the two copies of your database quietly disagree about where to send the parcel.

Same code. Same data. Same instant. The difference is architecture — and specifically, it comes down to a single question that most engineers never think to ask out loud:

How many machines are allowed to say yes to a write?

That's it. That's the whole lesson. The three famous replication topologies — leader-follower, multi-leader, leaderless — are not three different ways of copying bytes around. Copying bytes is the easy part. They are three different answers to that one question: exactly one node, a few designated nodes, or any node at all. Every other difference between them — the conflicts, the failover drama, the latency your users feel in Sydney — falls out of that choice like a consequence.

You've been set up for this. Back in Single Point of Failure — and Redundancy, the Antidote, the advice was blunt: don't have just one of anything that matters. But that lesson also flagged state as "the hard one," with a warning it didn't fully explain: duplicating state needs replication plus a single writer or consensus — or you get split-brain. This is where that sentence finally gets cashed.

In this lesson: the one question that generates all three topologies; how leader-follower actually works (we'll watch a real Postgres follower refuse a write, and then kill its leader); why anyone would accept the pain of multiple writers; how leaderless throws out the leader entirely; and the moment — which you'll drive yourself — when a conflict is born.

Scope: this lesson is about the shapes. Whether the leader waits for its followers before confirming a write, and what happens during the promotion when it dies, is Sync vs Async Replication & Failover. The overlap math that makes leaderless reads actually see recent writes is Quorums: N, R, W. What to do with a conflict once you have one — last-write-wins, vector clocks — is Version Conflicts. And the vocabulary for exactly how stale a read may be is The Consistency Spectrum. Here, we're just deciding who gets to say yes.

Replication itself is not complicated. You have data on one machine; you want it on three. You copy it. Machines are good at copying.

The complexity — all of it — arrives the moment the data changes. Now you have to decide where a change is allowed to originate. And there are only three possible answers:

• Exactly one node may accept writes. Everyone else is a read-only copy. This is leader-follower (you'll also hear single-leader, primary-replica, master-slave in older docs).
• A few designated nodes may accept writes, and they replicate to each other. This is multi-leader.
• Any node may accept a write. This is leaderless.

One, some, or any. That's the entire design space, and everything else is downstream of it.

Here's why that framing is worth more than three diagrams. With one writer, the question "which of these two writes happened first?" always has an answer — the order the leader put them in. There is a single place where time is decided. The moment you allow a second writer, that guarantee evaporates: two nodes can each accept a write to the same row at the same microsecond, and neither one knows the other exists yet. There is no global order, because there is no longer a single place where order is decided.

So a conflict isn't a bug you introduced. A conflict is a structural consequence of having more than one writer. You'll see the exact instant one is born later in this lesson, and I want you to notice that nothing goes wrong when it happens. Both writes succeed. Both machines are correct. That's what makes it hard.

Start with the answer almost everyone should use, and the one you're already running: exactly one node accepts writes.

Every write goes to the leader. The leader does the work, and then it ships its log to the followers, who replay it. If that sounds familiar, it should — it's the write-ahead log from WAL & Durability: Why Committed Data Survives, and that lesson promised you'd meet it again: the same log that heals a crash also streams to replicas. Here it is. Replication, in a leader-follower system, is log shipping. There's no second mechanism.

I built a real primary and a real standby to show you, rather than draw it. On the primary:

A walsender on the leader, a walreceiver on the follower, and the WAL flowing between them. Ask each machine what it thinks it is and you get the topology straight from the horse's mouth: pg_is_in_recovery() returns false on the leader and true on the follower. A follower is permanently "in recovery" — it spends its entire life replaying somebody else's log.

Now the part that makes this concrete. What happens if you try to write to a follower?

There it is. You don't politely configure single-writer as a convention your team agrees to respect — the database refuses. That error message is the topology. One node says yes; everyone else says no, on principle.

What you get for that restriction is enormous: conflicts are not resolved, they are impossible. There is exactly one place where write order is decided, so there is nothing to reconcile, ever. That is a colossal simplification, and it's why leader-follower is what Postgres, MySQL, Oracle, SQL Server, MongoDB, and Kafka all do by default. If you don't have a specific reason to do something else, this is the answer.

And it buys you read scaling honestly: reads can be served by any follower, so you can bolt on replicas and serve a lot more traffic.

But it charges you for it, in two ways — and you should feel both.

Cost one: the follower is behind, and that's not a bug.

Replication is asynchronous by default: the leader commits and confirms to your user immediately, and the followers catch up shortly after. On my localhost pair the lag was sub-millisecond (replay_lag: 983 µs), which sounds like nothing. So let me show you what that same architecture does when the lag is real. I paused replay on the follower — which is exactly what a slow or distant network does to you — and then wrote to the leader:

Your user saved her address, the write genuinely succeeded, and then her next page load happened to be routed to a follower — which cheerfully served her the old one. She thinks the save failed. She saves again. Now you have a support ticket and a confused customer, and nothing in your system is broken.

That's the tax on read scaling: the copies you added to serve reads are copies of the recent past. (Precisely how stale a read is allowed to be — and the guarantees with names, like read-your-writes — is The Consistency Spectrum. For now, just know the staleness is real and it has your customer's address in it.)

Cost two: kill the leader and all writes stop. Everywhere.

I killed the primary container and asked the survivor two questions.

Read this carefully, because it's the honest price of the simplicity you just bought. The leader is dead. The follower has all the data, is perfectly healthy, and is sitting right there — and it will not take your write. It doesn't matter that it could. It's a follower. Followers say no.

Your reads are fine and your writes are simply down — not degraded, not slow, down — until somebody promotes that follower to leader. Which raises every question you're now asking: who decides it's dead? How long do we wait? What if we're wrong? All of that is the next lesson, Sync vs Async Replication & Failover, and it's a genuinely nasty problem. I'm stopping here on purpose.

So: one writer gives you no conflicts and a write outage. That trade is usually correct. Sometimes it isn't.

Here's a number that ruins single-leader for certain systems: a round trip between AWS us-east-1 (Virginia) and eu-west-1 (Ireland) is about 68 milliseconds.

Now put your one and only leader in Virginia. Every write from a European user pays that 68 ms before your database has done a single microsecond of work. Add a few round trips for the connection and the transaction and your Dublin customer is waiting a fifth of a second to change a checkbox — for reasons that have nothing to do with your code and everything to do with the speed of light and a very long cable.

Multi-leader deletes that number. Put a leader in each region. European writes land in Ireland in about a millisecond; American writes land in Virginia. The leaders replicate to each other in the background, asynchronously. Your users stop waiting on the Atlantic.

There are two other situations where multi-leader isn't a choice so much as a description of reality:

• Clients that work offline. The calendar app on your phone accepts writes on a plane with the wifi off. That phone is a leader — it took the write, it's authoritative for a while, and it will sync later. (CouchDB was designed around exactly this.)
• Collaborative editing. Every person in the document is writing to their own local copy in real time. That's a multi-leader system whether you called it one or not.

So multi-leader buys you write locality and write availability. Now let's talk about what it charges.

Back to Frankfurt and Virginia, 14:02:31, both updating the same shipping address.

Under one leader, this is boring. Both writes travel to the same machine, that machine puts them in some order, the second one wins, and the story ends. Somebody's edit was overwritten — but the database is consistent, and it can tell you exactly what happened.

Under two leaders, watch what actually occurs:

1. Frankfurt's write arrives at the European leader. It's valid. It commits. The user gets a success.
2. Virginia's write arrives at the American leader. It's also valid. It also commits. That user also gets a success.
3. Neither leader has any idea the other one just did that. They haven't spoken yet.
4. A moment later, replication catches up — and the two leaders discover they hold two different, equally committed, equally correct values for the same row.

That is a conflict. And I want you to notice everything that did not go wrong. No network failed. No code raced. No transaction was violated. Both databases behaved perfectly. The conflict was not caused by a bug — it was caused by the topology. You allowed two machines to say yes, and two machines said yes.

This is also what makes it genuinely nasty in production: the writes succeed at write time, and the conflict is only detected later, asynchronously, when the replication streams meet. There is no exception for your application to catch. You find out in a reconciliation job, or in a support ticket.

So the honest way to describe multi-leader is this: a conflict is not a failure, it's the receipt. It is what you agreed to pay when you bought local writes and write availability. The question multi-leader forces on you is not how do I avoid conflicts — you can't, that's the deal — but how do I decide who wins?

That question is a whole lesson of its own (last-write-wins and why it quietly loses data, vector clocks, CRDTs), and it's Version Conflicts. Today, it's enough to know the bill exists and that you signed for it.

One more thing worth saying out loud, because it's the most common reason people reach for multi-leader and it's usually wrong: multi-leader does not buy you write throughput. Every leader still has to apply every other leader's writes — that's what keeping the copies in sync means. So each node ends up doing roughly the same total write work it would have done anyway. Multi-leader is a latency and availability tool, not a throughput tool. If writes are your bottleneck, replication is not your answer — partitioning is, and that's a later section.

PostgreSQL, for what it's worth, ships no native multi-master at all, deliberately. You have to reach for third-party tooling. Their reasoning is basically this section: conflicts aren't resolved for you, divergence is easy, and debugging it is miserable.

The third answer is to stop pretending anyone is in charge. Any node accepts a write.

In a leaderless system — the design Amazon published as Dynamo, and which Cassandra, Riak, Voldemort and ScyllaDB grew out of — the client doesn't look up a leader, because there isn't one. It sends its write to several nodes at once. Later, when it reads, it asks several nodes at once and compares the answers it gets back.

The payoff is availability, and it's a big one. There is no leader to lose, so there is no failover step at all. Remember what happened when I killed the Postgres leader — every write in the system stopped until a human intervened. In a leaderless system, a dead node is barely news: the write simply goes to the nodes that are up. That's the entire point of the design.

The cost is that the tidy illusion of one database is gone. Different nodes can hold different values, and now the client — or a coordinator acting for it — has to make sense of what comes back from several machines that don't fully agree. The problem didn't disappear when the leader did. It moved.

Which raises the obvious question: if I write to some nodes and read from some nodes, how do I guarantee I actually see my own write? There's a beautiful bit of overlap arithmetic that answers that, and it's the next-but-one lesson, Quorums: N, R, W. And how the nodes quietly heal their disagreements in the background is Read Repair & Hinted Handoff. Both are exactly the machinery that makes 'any node says yes' actually workable — and both are somebody else's chapter.

For now, hold the shape: one node, some nodes, any node.

Reading about a conflict being born is one thing. Causing one is another.

Below is a live three-region system. Pick a topology, then send writes by clicking a node and typing a value — watch replication actually flow between the regions. Kill nodes. Break things.

Try this first, because it's the whole lesson in ten seconds: stay on leader-follower and try to write to a follower. It will refuse you, exactly like the real Postgres did. Now kill the leader and try to write anywhere at all — you can't, and that's the real cost of one writer. Then switch to multi-leader, write a different address to Frankfurt and to Virginia before letting them sync, and watch a conflict come into existence. Notice that both writes succeeded. Notice that nothing broke.

Now we can finally cash the warning from Single Point of Failure — and Redundancy, the Antidote: duplicating state needs replication plus a single writer or consensus — or you get split-brain.

Here's what that actually means, and it's the best punchline in this lesson.

You run a sensible leader-follower system. One writer. No conflicts possible. Then the network partitions: the followers can no longer hear the leader. From where they're standing, the leader is dead. So — doing exactly what you configured them to do — they promote one of their own. There is now a new leader taking writes.

Except the old leader isn't dead. It's just unreachable from over there. It's still up, still healthy, still happily accepting writes from whichever clients can still see it.

You now have two nodes accepting writes to the same data. Which is to say: you are running a multi-leader system. You just didn't mean to, and — this is the part that hurts — you have none of the conflict machinery a real multi-leader system would have built. No conflict detection, no resolution policy, no reconciliation. Two databases are diverging in silence.

That's split-brain. It isn't some exotic distributed-systems demon. It's a single-leader system that accidentally became a multi-leader system — and that is precisely why C1 told you that duplicating state needs "a single writer or consensus." Consensus is the machinery that stops two nodes from ever both believing they're the leader.

How you actually prevent it — fencing, quorums, leader election — belongs to Sync vs Async Replication & Failover and to the consensus lessons later on. What you need today is the diagnosis, because it explains why the failover problem is hard rather than just fiddly: the danger isn't losing your leader. It's ending up with two.

And if you're now asking the obvious follow-up — fine, but during a partition, what* should** the system do? Keep taking writes on both sides and sort it out later, or refuse writes until it's sure who's in charge?* — then you have just independently discovered the question that CAP, Properly exists to answer. Hold it. You'll get there in four lessons, and you'll get there having already felt why it's a real dilemma rather than a slogan.

If you take one decision procedure from this lesson, take this one.

Start with leader-follower. It is the right answer far more often than the internet's enthusiasm for distributed systems would suggest. It makes conflicts impossible, which is a guarantee no other topology can offer at any price. Accept that you'll need read replicas for read scale, that those replicas serve slightly stale data, and that you'll need a real failover plan. Almost every company you admire runs this.

Move to multi-leader only when geography or physics forces you. Users on another continent eating 68 ms per write; clients that must work offline; documents several people edit at once. Those are real reasons. "We want more write throughput" is not one — you'd be buying conflicts and getting no throughput. And go in with your eyes open: you are signing up to answer who wins for every row in your database.

Choose leaderless when availability of writes is the product. Shopping carts that must accept an add-to-cart even while the datacentre is on fire; telemetry that must never reject an event. You are trading a coherent single view of your data for the promise that a write is almost never refused.

And whichever you pick, say the trade out loud. That's what separates "we use Cassandra" from "we chose leaderless because a rejected write costs us more than a temporarily disagreeing one — and we accepted that our clients have to reconcile."

You can stand up a real leader and a real follower in about three minutes, and the two commands that matter will teach you more than any diagram. Everything below is copied from the run in this lesson.

Start a primary with replication switched on, take a base backup of it into a second container (pg_basebackup -R writes the follower's connection config and the standby.signal file that tells it what it is), and start that second node.

Now the two experiments that matter. Predict each before you run it.

Experiment 1 — try to write to the follower. What do you think happens?

Experiment 2 — make the follower serve you stale data on purpose. Pause its replay, write to the leader, then read from both. This is the read-your-writes problem, manufactured in ten seconds:

Experiment 3 — kill the leader (docker kill pri) and then try to write to the follower. It still refuses. Reads work; writes are gone. Sit with that for a second — the machine that could obviously serve the write is choosing not to, because that is what a follower is. That feeling is why the next lesson exists.

• "Replication is a backup." No. Replication faithfully copies your DROP TABLE orders to every replica in milliseconds. Replication protects you from a machine dying. Backups protect you from yourself. You need both, and they are not substitutes.
• "Adding replicas makes writes faster." No — in leader-follower, every write still goes through one node. Replicas scale reads. If writes are your bottleneck, replication is the wrong tool entirely.
• "Multi-leader gives me more write throughput." Mostly no. Each leader must still apply every other leader's writes, so each node does roughly the same total write work. You bought locality and availability, not throughput — and you paid in conflicts.
• "Leaderless means no leader, so no problems." It means the problems moved to the client, which now talks to several nodes and has to make sense of disagreeing answers.
• "A conflict means something is broken." No — a conflict is the structural, expected consequence of letting more than one node say yes. Both writes succeeded. Nothing failed.
• "My follower has the data the moment I commit." No — replication is asynchronous by default. The follower in our run was serving a stale address while 136 bytes behind.
• "Split-brain is an exotic edge case." It's the most ordinary thing in the world: a single-leader system that accidentally ended up with two leaders.

• One question generates all three topologies: how many nodes may say yes to a write? One (leader-follower), some (multi-leader), any (leaderless). Everything else is downstream.
• Replication is log shipping. The leader ships its WAL; the follower replays it forever (pg_is_in_recovery() is true on a follower for its entire life). It's the same log that survives a crash.
• The topology is enforced, not agreed. A follower answers a write with ERROR: cannot execute INSERT in a read-only transaction. That line is leader-follower.
• One writer = no conflicts, ever — and a write outage when the leader dies. Reads survived our kill; writes did not, and no healthy follower would take them.
• Followers are behind, by design. Ours served a stale address while 136 bytes behind. Read replicas are copies of the recent past.
• Multi-leader is a latency/availability tool, not a throughput tool. It exists because us-east-1 → eu-west-1 is ~68 ms. It charges you in conflicts, and the conflict is architectural, not a bug: both writes succeed, and nobody finds out until later.
• Leaderless moves the problem to the client — and buys write availability by never needing a failover.
• Split-brain is single-leader accidentally becoming multi-leader — which is exactly why duplicating state needs "a single writer or consensus."

Next — Sync vs Async Replication & Failover: the leader is dead, a healthy follower is sitting right there, and someone has to decide whether to promote it. Wait too long and you're down. Move too fast and you've got two leaders. That's the job.