Sync vs Async Replication & Failover

Five customers paid. Each one clicked the button, the database committed, and each one got back a cheerful "Payment saved." Their money had moved. As far as they, your API, and your logs were concerned, those transactions were done.

Ninety seconds later the leader's hardware died. The failover worked exactly as designed — the standby was promoted, traffic rerouted, the site came back up in under a minute. A textbook recovery.

And the database that is now in charge has never heard of those five payments. No error was raised. Nothing is corrupt. No alert fired. The rows are simply not there, and they never will be.

I didn't make that up as a scare story — I ran it, and here is the count from the actual experiment:

That gap — the writes your database said yes to but never handed to anybody else — is called the loss window, and every asynchronous replication setup on earth has one. Including, almost certainly, yours.

The previous lesson, Replication Topologies, left you standing over a dead leader with a perfectly healthy follower that refused to take your write, and I promised the answer was here. It is. But the answer turns out to be two questions, not one — and they're really the same question, asked at two different moments:

How long are you willing to wait?

Ask it at commit time and you get the commit dial: does a write count as saved the instant the leader has it, or only once a follower has it too? Ask it at failure time and you get the failure-detector dial: how long do you wait, staring at a leader that has stopped answering, before you declare it dead and promote somebody else?

Turn either dial up and you get safety and slowness. Turn either down and you get speed and risk. There is no setting that gives you everything, and this lesson is about learning to say out loud which one you chose.

In this lesson: the commit dial, notch by notch, with the real cost of each measured; the trap that makes people think they're synchronous when they aren't; the counter-punch nobody expects (turn synchronous replication on, kill the follower, and watch your leader stop accepting writes); the failover sequence and why the detector dial cuts both ways; and GitHub's October 2018 outage, where every mechanism in this lesson fired correctly and produced 24 hours of pain.

Scope: how the surviving nodes actually agree on who becomes the new leader — the election, the votes, the consensus algorithm — is deliberately not here. It's a genuinely deep topic and it gets its own treatment later in the course. Today, promotion is a thing that happens; what matters is what it costs you. Quorum arithmetic is Quorums: N, R, W, and the vocabulary for how stale a read may be is The Consistency Spectrum.

Everything in this lesson hangs off two settings, and they are the same idea wearing different clothes.

The commit dial. A write arrives. The leader can tell the user "saved" right now — fast, but if the leader dies before the follower gets that write, it's gone. Or it can hold the user's request open until a follower confirms it has the data — slow, but now the write survives the leader's death. Every millisecond you make the user wait buys you a little more certainty.

The failure-detector dial. The leader stops answering heartbeats. Is it dead, or is it just busy? A garbage-collection pause looks exactly like a dead machine from the outside. Promote too eagerly and you'll promote a leader that was never dead — now you have two. Wait too politely and you're just… down, watching a healthy follower do nothing.

These two have proper names, and using them will make you sound like someone who has been paged at 3am:

• RPO — Recovery Point Objective: how much data may I lose? The commit dial sets this. RPO=0 means "not one acknowledged write, ever," and that requires synchronous replication.
• RTO — Recovery Time Objective: how long may I be down? The failover machinery sets this — detection plus promotion plus rerouting.

They are not the same dial, they trade against different things, and — this is the part people get wrong — tightening one often loosens the other. Crank RPO to zero with synchronous replication and you've just made your leader's availability depend on a follower's health. We'll watch that happen.

Let's take the commit dial apart first, because it's the one you can measure.

You've met this dial before, in passing. WAL & Durability: Why Committed Data Survives mentioned that synchronous_commit is "a dial with a loss window" and then moved on, because at the time you only had one machine and the loss window was tiny. Now that there's a second machine in the picture, that one throwaway line turns out to be the most consequential setting in your database. So let's actually turn it.

In Postgres this dial is a single setting called synchronous_commit, and it has five notches. Each one is a different answer to what has to be true before I tell the user "saved"?

• off — Don't wait for anything. Not even the local disk. Fastest possible; there's a loss window even with no replicas at all.
• local — Wait for the write to hit this machine's disk. Do not wait for any follower. This is what people mean by "asynchronous replication."
• remote_write — Wait until the follower has received the record and handed it to its operating system. Survives the follower's database crashing; does not survive the follower's machine losing power, because the bytes may still be sitting in an OS buffer.
• on (the default) — Wait until the follower has flushed it to durable storage. Now it survives the follower's machine dying too.
• remote_apply — Wait until the follower has actually applied it, so a query on the follower would see it. The strongest, and the slowest, because it waits for replay.

That progression isn't arbitrary — it's a ladder of "how far into the follower must this write get before I'll take responsibility for it?" And each rung costs you. Here's what it actually costs, measured over 500 real commits on a real primary/standby pair:

Every rung of that ladder is a purchase. You are buying durability with latency, in milliseconds, on every single write your system will ever do. Whether that's a bargain or a catastrophe depends entirely on what the write is — which is the honest answer to "what should I set it to?" A payment and a telemetry ping do not deserve the same guarantee, and Postgres will happily let you set this per transaction.

Now, before you go and switch it to on and feel safe — there's a trap.

Setting synchronous_commit = on does, by itself, nothing at all for replication.

It's a genuinely nasty piece of API design, and I walked straight into it while building the experiment for this lesson. Here's the live output from a primary with a healthy, streaming standby attached:

synchronous_commit is on. The standby is connected and streaming. And the replication is asynchronous — because Postgres has no idea which standby it's supposed to wait for. Until you name one in synchronous_standby_names, the strong-sounding settings (on, remote_write, remote_apply) all quietly collapse into local-only behaviour.

You have a config that reads synchronous, a monitoring dashboard that says replicating, and an RPO of whatever the network felt like today. Name the standby and it changes:

If you take one operational thing from this lesson: go and check sync_state in pg_stat_replication on your production primary right now. Not the setting you think you set — the state the database is actually in. It is the difference between RPO=0 and RPO="we'll find out during the incident."

Right. Synchronous replication is properly on now. RPO is zero. No acknowledged write can ever be lost. This feels like the correct, professional, grown-up setting, and you may be wondering why anyone would run anything else.

So let's kill the follower. Not the leader — the follower. The spare. The one that isn't serving any traffic.

The leader is completely healthy. It has a working disk, plenty of RAM, and nothing wrong with it whatsoever. Watch:

The leader is alive. It's answering reads instantly. It just will not finish a COMMIT — and it would have hung there forever if I hadn't killed the client.

And it's right to. You told it: never acknowledge a write until a follower has it. There is no follower. So there is no acknowledgment. The database is keeping precisely the promise you asked it to keep.

You killed a replica and your primary stopped accepting writes. That is the true price of RPO=0, and it catches people because it feels backwards — you added a machine for redundancy, and instead you added a machine that can take you down. The write path now depends on both nodes being alive. You made durability stronger and availability weaker, with one setting.

This is also why real systems have an escape hatch: when the synchronous standby is lost, many setups fall back to asynchronous so writes can keep flowing. Which is a completely reasonable engineering decision, and you should be clear-eyed about what it means: your RPO=0 guarantee holds only while nothing is wrong. The moment you actually need it, it degrades to the thing it was supposed to protect you from.

(There's a middle road, and it's what most large deployments actually run: require an acknowledgment from any one of several followers rather than one specific machine. Then losing a single follower doesn't stall the leader. That is the same overlap idea you'll meet properly in Quorums: N, R, W.)

So most people run asynchronous replication — local, or on with nobody named — because it's fast and because a dead replica shouldn't take down the site. That is a defensible choice, made by serious engineers, and it is almost certainly what you are running.

Here is the bill.

The setup is aggressively ordinary. The standby is down or lagging — maybe it's being patched, maybe the network is having a bad afternoon. Nobody is paged; async replication is supposed to tolerate this. The leader keeps working, because that's the entire point of async.

Five customers pay. Every COMMIT returns success. Every customer sees "Payment saved."

Then the leader's hardware dies, and we do the thing we practised: promote the standby.

Zero of five. The failover was a success — pg_promote() returned true, the new leader is out of recovery, the service is up, the dashboard is green. And five people who were told their payment went through are, as far as your system is now concerned, people who never paid.

Sit with how quiet that is. Nothing raised an error. No corruption, no exception, no alert. There is no log line anywhere in your infrastructure that says "by the way, we discarded eight acknowledged transactions." The writes lived only in the leader's memory and disk, the leader is gone, and with it the only record that they ever happened.

That set — acknowledged by the leader, never received by the follower — is the loss window. Its size is exactly your replication lag at the moment of death. Async replication doesn't risk this. Async replication is this, and it works exactly as designed. The question was never whether you have a loss window; it's whether you have decided how big it's allowed to be, and whether the things inside it are telemetry pings or somebody's rent.

We've been saying "promote the standby" as if it's a button. It's four steps, and every one of them can hurt you.

1. Detect. Something must decide the leader is dead. In practice: it stopped answering heartbeats for N seconds. That's it. That's the whole basis on which you're about to make an irreversible decision.

2. Fence. Make certain the old leader can no longer accept writes. Not "believe it's dead" — make it dead, or cut it off from the network, or revoke its access to storage. This step has the least subtle name in all of computing: STONITH — Shoot The Other Node In The Head. Skip it and you will eventually have two leaders, which is the split-brain from Replication Topologies.

3. Promote. A follower stops replaying the log and starts writing its own. In Postgres that's pg_promote(); afterwards pg_is_in_recovery() flips from true to false, and it's a leader.

4. Reroute. Clients must actually start talking to the new leader — DNS, a proxy, a service registry, a connection pooler. Until this happens, nothing is fixed. A promoted database that no traffic reaches is just an expensive idle machine, and this is the step that most often turns a 30-second failover into a 20-minute outage.

How the surviving nodes agree on which follower gets promoted — the election, the votes, the algorithm that stops two of them promoting themselves simultaneously — is a deep and genuinely beautiful problem, and it is not this lesson. It gets a proper treatment later. What you need today is to know that this step must be agreed, not merely attempted, and that agreement under a network partition is the hardest thing in distributed systems.

Now the dial. How long should step 1 wait?

Too long and you're simply down. The leader died at 14:00:00, the detector is patient, and you spend ninety seconds serving errors to everyone while a healthy follower sits there doing nothing.

Too short and you get something much worse than downtime: a false positive. A stop-the-world garbage collection pause, a saturated network link, a slow fsync — from the outside, a machine that is merely busy is indistinguishable from a machine that is dead. Your detector fires, you promote a follower, and now you have two leaders, both taking writes, diverging with every second. You didn't recover from a failure; you manufactured one.

This is why production heartbeat timeouts sit in the unglamorous 1–5 second range. Not because someone was lazy, but because that's the honest middle of a genuine trade. (There's even a folklore failure mode for getting the fencing wrong in a two-node cluster: the STONITH deathmatch, where each node believes the other is broken, shoots it, and the victim reboots and shoots back. Forever.)

This is a lesson about two settings whose consequences only show up at the worst possible moment, so let's bring that moment forward.

Below you run a leader and a follower. Set the commit dial. Set the detector timeout. Send some payments — real ones you type — and watch which of them make it across to the follower. Then kill the leader and run the failover.

Try this first: leave it on async, let the follower fall behind, send three payments, and then kill the leader. Look carefully at what the promoted follower has. Those are real customers who were told 'saved'. Now switch to sync and do the exact same thing — nothing is lost. Then, while still on sync, kill the follower instead and try to write. Notice which machine stops working, and ask yourself whether you expected that.

Finally, drag the detector timeout down to something aggressive and make the leader pause rather than die. Congratulations: you have two leaders.

On 21 October 2018, someone replaced a piece of failing 100Gb optical equipment on GitHub's network. The swap caused a 43-second loss of connectivity between GitHub's US East Coast network hub and its primary East Coast data centre.

Forty-three seconds. Read the rest of this knowing that number.

GitHub ran Orchestrator to manage MySQL topology and automate failover. During the partition, Orchestrator nodes on the West Coast and in the public cloud could still see each other, formed a quorum, concluded — correctly, from where they were standing — that the East Coast primary was unreachable, and failed the clusters over to the West Coast. The automation did exactly what it was built to do.

But during those 43 seconds, the East Coast primary was still up. It was still accepting writes from clients that could still reach it. And because replication is asynchronous, those writes were never sent west.

When the network came back, GitHub had two clusters, each holding writes the other had never seen. That's the loss window and split-brain in the same breath. And there was a nasty second-order effect: the new primary was now on the West Coast, while most of the applications were on the East — so every database call was suddenly paying a cross-country round trip, exactly the latency tax we measured in Replication Topologies.

A 43-second network blip produced more than 24 hours of degraded service. The site was fully restored the following evening.

Here's what makes this the perfect case study, and it's not schadenfreude. Nothing malfunctioned. The network fault was brief and real. The failure detector correctly observed an unreachable primary. The failover automation correctly promoted a replica. Async replication correctly did not wait. Every single component behaved exactly as designed — and the composition of those correct behaviours produced a day-long outage.

That is the lesson. These dials don't fail by breaking. They fail by working, in a combination nobody sat down and chose on purpose.

There is no configuration that gives you fast writes, zero data loss, and a leader that shrugs off its replica dying. Pick two. Say which two out loud, and write it down somewhere your future on-call self will find it.

Async (the default, and usually right). Fast writes, the leader survives replica failures, and you accept a loss window the size of your replication lag. Correct for: analytics, telemetry, sessions, feeds, most product features. Be honest that RPO > 0 and go find out how big it actually is — your lag is your exposure.

Sync (when losing a write is worse than a slow write). Payments, ledgers, orders, anything where a customer has been told something irreversible happened. Price it: ~2.4× the commit latency in our run, and your leader now depends on a follower's health. Mitigate that by requiring any one of several followers rather than one named machine — never by quietly degrading to async and keeping the RPO=0 line in your architecture doc.

And you can mix them. This is the move most teams miss: synchronous_commit is settable per transaction. Your payment writes can be synchronous while your click-tracking writes are off, in the same database, in the same minute. The dial does not have to be set once, globally, forever — the data should decide.

On the failover dial: be suspicious of aggressive timeouts. Every second you shave off detection buys RTO and sells you false-positive risk, and a false positive is worse than the outage it was trying to prevent. Fence properly. And test the reroute step, because promotion is the part that works in the drill and rerouting is the part that doesn't.

The one thing you must not do is have no answer. "We use the defaults" is a decision — it's just one nobody made deliberately.

The most valuable ten minutes you can spend after this lesson is making your own database lose your own data, on purpose, where it doesn't matter. Everything below is from the run in this lesson.

1. Find out whether you're actually synchronous. Run this on a real primary — including, if you're feeling brave, production:

2. Measure your own commit ladder. Time a few hundred separate commits at each notch. Predict the ordering before you run it — you already know it must be monotonic, but how much it costs you on your hardware and your network is a number only you can get:

3. Make your leader hang by killing your follower. This is the one that changes how people think, and it takes about thirty seconds. Predict what happens before you run it:

4. Lose data on purpose. Go async, stop the standby, insert five clearly-labelled rows (all of which will commit successfully), kill the primary, then start the standby and SELECT pg_promote(). Count your rows. Then go and look at whatever synchronous_commit your actual production database is running, and decide whether you're comfortable.

(If you'd rather do this properly, with every command spelled out and a replica you build from scratch, there's a full terminal walkthrough waiting for you later in this section: 🧪 Codelab: Postgres Replication + Failover. This is the version you can do in a coffee break.)

• "Synchronous replication means zero data loss." Only if it's genuinely configured (synchronous_standby_names — the trap), only while the sync standby is alive, and only if your system doesn't quietly degrade to async when it isn't. A guarantee that evaporates under stress is not a guarantee.
• "We have automatic failover, so we're covered." GitHub's automatic failover is precisely what turned a 43-second network blip into 24 hours. Automation makes the decision faster, not wiser.
• "The replica has my data." Not the writes in the loss window. We measured eight acknowledged rows — five of them customer payments — that the promoted replica had never heard of.
• "A more aggressive failure detector is safer." Backwards. A short timeout manufactures false failovers, and a false failover gives you two leaders — strictly worse than the downtime you were avoiding.
• "RPO and RTO are roughly the same thing." RPO is how much data you may lose; RTO is how long you may be down. Sync replication tightens RPO and hurts availability. A fast detector tightens RTO and risks split-brain. They pull in different directions.
• "Failover is done once the new leader is promoted." It's done when clients are actually talking to it. Promotion without rerouting is a database nobody is using.
• "The old leader is dead, so it can't cause trouble." Until it wakes up. That's what fencing is for, and it's why the acronym is STONITH.

• Two dials, one question — how long are you willing to wait? The commit dial sets RPO (how much data you may lose). The failure-detector dial sets RTO (how long you may be down). They trade against different things and often against each other.
• The commit ladder is real and it is monotonic (measured, 500 commits each): off 0.706 ms → local 1.154 → remote_write 1.356 → on 1.728 → remote_apply 2.392. 3.4× from end to end. No free notch.
• synchronous_commit alone does nothing. With synchronous_standby_names empty, sync_state is async no matter what the setting says. Go check the real state, not the intended one.
• Sync replication makes your leader depend on your follower. Kill the follower and the healthy leader stops accepting writes (measured: an 8,101 ms hang; reads unaffected). RPO=0 costs you availability, and "fall back to async" quietly refunds the guarantee.
• Async replication has a loss window, always. It's exactly the size of your replication lag. We watched a clean, successful failover discard 8 acknowledged rows — 5 of 5 customer payments — with no error, no corruption, and no alert.
• Failover is four steps: detect → fence → promote → reroute. Skip fencing and you get two leaders. Forget rerouting and you have a promoted database nobody is talking to.
• The detector dial cuts both ways: too slow is downtime; too fast is a false failover, which is worse. A busy machine and a dead machine look identical from the outside.
• GitHub, October 2018: every component worked exactly as designed, and 43 seconds of network trouble became 24 hours of degraded service. These dials don't fail by breaking. They fail by working, in a combination nobody chose.

Next — Quorums: N, R, W: we've been treating "the follower confirmed it" as one machine saying yes. But if you have five replicas, how many must answer before a write counts — and how many must you ask before a read is trustworthy? There's a piece of overlap arithmetic that makes leaderless systems work, and it's a genuinely lovely idea.