Network Partitions & Split Brain

On the evening of October 21, 2018, a maintenance crew was replacing a piece of failing optical equipment on GitHub's network. For 43 seconds, the link between GitHub's East Coast network hub and its primary East Coast datacenter went dark. Routine stuff; connectivity came right back.

The recovery took 24 hours and 11 minutes.

Nothing crashed. No disk died, no process was killed, no data center burned. What happened in those 43 seconds is the subject of this lesson: the automated failover system, following its rules correctly, promoted a new set of database primaries on the West Coast while the East Coast primaries were still alive and still taking writes. When the link healed, each coast held rows the other had never seen. Two histories. One brand.

The last lesson ended on exactly this cliff: a failure that isn't a node at all. The network splits, both halves are perfectly healthy, and each concludes the other is dead. This lesson is about what happens next — the birth of a second leader, the price of two histories, and the two defenses that keep one of them from being yours: the quorum tiebreak and the fencing token.

Scope: how the survivors vote for a new leader is deliberately left sealed — that machinery is Leader Election and Raft, Visually, later in this course, exactly as promised back in Sync vs Async Replication & Failover. And the timers that decide "he's dead, promote" get their own lesson next (Heartbeats & Failure Detection). Today: the split, the damage, the tiebreak, the zombie, the fence.

You've already met partitions twice, from two angles. CAP, Properly proved that one dropped packet is, formally, a partition, and showed a single isolated node choosing between answering and refusing. And the first lesson of this course measured how often the substrate fails — dozens of link failures a day inside one well-run fleet, where redundancy cut the impact only ~40%.

Here's the cluster-level view of the same event. A partition doesn't remove machines; it redraws the map. One healthy cluster becomes two smaller, fully-functional clusters, each able to talk among itself and completely deaf to the other. And now apply the bedrock fact from Welcome to Hard Mode: neither side can distinguish "the others died" from "we got cut off." The evidence is identical — silence.

So both sides do the reasonable thing. Both conclude they are the survivors. Both carry on serving users. That is the trap in its purest form: a partition doesn't take your system down. It takes your system in two. Everything stays up, which is precisely the problem — because some of what's running now exists twice.

Split brain is not caused by chaos. It's caused by your own safety machinery, working as designed, on the wrong side of a cut.

Recall the failover playbook from Sync vs Async Replication & Failover: detect → fence → promote → reroute. Now run it during a partition. The replicas on the far side of the cut stop hearing the leader's heartbeats. Their detector fires: he's dead. They promote one of their own, reroute their half of the traffic to it, and get back to work. Textbook.

Except the old leader isn't dead. It's on the other side of the cut, healthy, serving its own half of the traffic, wondering why half its followers went quiet. Nobody told it about the funeral. Nobody could; that's what a partition means. You now have two nodes, each wearing the crown, each accepting writes, each convinced the other is gone.

This is exactly the scene GitHub's postmortem describes. Their failover coordinator held a quorum of votes on the West-Coast side and promoted new primaries there — correctly, by its own rules — while East-Coast databases were still alive. And it's the moment Replication Topologies warned about with a rule we can finally cash in full: duplicating state needs "a single writer or consensus — or you get split-brain." Today is the lesson where you watch the single writer become two.

While the link is down, the two leaders diverge. Ana's cart gets a hat on the east side; her payment lands on the west side; a username is registered twice; two different rows both hold order #48291. Every write is locally legitimate — acknowledged, durable, committed. You saw this shape in Replication Topologies: two leaders holding "two different, equally committed, equally correct values for the same row."

Then the network heals, and here is the ugly truth: the partition ends; the divergence doesn't. Reconnecting the halves doesn't merge two histories any more than plugging two laptops into the same router merges their disks. Someone must now decide, row by row, which truth survives — and the naive resolver, last-write-wins, silently destroys fully-committed data, as Version Conflicts demonstrated with a timestamp.

GitHub's numbers make the cost concrete. The cut lasted 43 seconds. But with unique writes stranded on both coasts, they had to choose: fail back east and abandon the West-Coast writes, or fail forward west and let East-Coast apps eat a cross-country round trip on every write. They failed forward to protect the data, and spent a day paying for it. The outage is cheap. The divergence is expensive. Which is why every defense in this lesson aims at the same target: never let the second history start.

The first defense is one you already own the math for. Quorums: N, R, W taught that two majorities of the same cluster must overlap — which means there can only ever be one side holding a majority. That arithmetic is the tiebreak.

The rule: a node may act as leader only while it can reach a majority of voters. Cut a five-node cluster 3-and-2, and the three-side keeps (or promotes) a leader while the two-side, unable to muster a majority, steps down — refuses writes, serves nothing it can't vouch for, and waits for the network to come home. No second history is ever born. You've seen the spirit of this before: in CAP, Properly, choosing consistency during a partition took the minority down, not the system.

Two practical consequences fall straight out of the arithmetic. Voter counts should be odd — split four voters 2-and-2 and neither side has a majority, so your tiebreak just took the whole system down. And the classic two-node HA pair is the worst possible shape: no majority exists to win, which is why production setups add a cheap third witness node whose only job is to break ties. (Elasticsearch spent years as the canonical cautionary tale here: its old minimum_master_nodes setting left the quorum floor to the operator, and clusters that got it wrong reliably elected two masters under stress. Modern versions build the quorum in — the tiebreak is too important to be a config knob someone can fat-finger.)

So: quorum on, minority steps down, one crown. Solved? Almost. There's one attacker the majority cannot outvote, because it doesn't know the vote is happening.

Rewind to a fact from the first lesson: a node can vanish without failing — a garbage-collection pause froze a perfectly healthy process past its timeout, and the cluster convicted it as dead. Now make that node your leader.

It pauses at 09:00:00, mid-flight, holding the crown. At 09:00:15 the detectors fire; the quorum, doing everything right, elects a successor; the world moves on. At 09:00:40 the old leader's pause ends. From its point of view, no time has passed. It was the leader a millisecond ago; it has writes queued; its lease felt valid when it last checked. So it does what leaders do: it writes.

Notice what just failed. There was no partition — the network was healthy the whole time. And quorum didn't help — the vote happened while the zombie was frozen; you cannot outvote a process that isn't listening. The failover lesson left you with exactly this chill: "The old leader is dead, so it can't cause trouble." Until it wakes up. That lesson's answer was the blunt one — STONITH, make it dead for real — which works when you can reach the box. But a paused process on a healthy machine, or a leader on the far side of a cut, may be un-shootable at the moment it matters.

What you need is a defense that doesn't require finding the zombie at all. Let it wake. Let it write. And have the write bounce.

The graceful fix moves the defense to the last place the zombie must pass: the resource itself.

Attach a number to leadership. Every time a new leader is promoted, the number increments — epoch 5, epoch 6, epoch 7. Every write a leader issues carries its number, and the storage keeps a single integer of state, the highest number it has ever seen. The rule it enforces is one line: never accept a write whose token is lower than the highest seen.

Run the zombie through it. Old leader, epoch 5, pauses. New leader is promoted with epoch 6 and writes — the storage notes: highest = 6. The zombie wakes and fires its queued write, stamped epoch 5. The storage compares 5 < 6 and rejects it cold. Nobody had to find the zombie, detect the pause, or win a race. Its authority simply expired arithmetically, and the one component it cannot bypass — the thing it's trying to write to — enforced the expiry.

This is the mechanism the failover lesson gestured at with "revoke its access to storage," made precise. The idea is due to a famous essay by Martin Kleppmann on distributed locking, and it's not hypothetical machinery: ZooKeeper hands out exactly such a number (its zxid), and well-built storage layers check it on every mutation. The cost is equally real, so say it honestly: fencing is invasive. Every write path must carry the token and every resource must check it — which is why so many systems skip it, and why the ones that skip it meet the zombie eventually. Note the pairing, because interviewers probe it: quorum stops the second history from starting; fencing stops the old history from coming back. You want both.

Time to hold the cable yourself.

Try this first: with no defenses armed, cut the link, send a few writes to each side, and heal — count the divergence you just created. Then arm the quorum tiebreak and do it again: watch the minority step down and one history survive. Now the part that separates this lesson from a diagram: leave quorum on, pause the leader, let the failover finish, let the new leader take a write, and wake the old one. Its stale write sails into your storage — until you arm fencing and the gate reads token 5, sees it has already accepted 6, and turns the zombie away at the door.

"Defend everything, everywhere" is not a design; it's a budget overrun. The senior move is to sort your state into two piles before the partition happens.

State that may fork — briefly, on purpose. A shopping cart, a like counter, a presence flag. Let both halves keep accepting it through a partition, and merge on heal with the conflict tools from Version Conflicts (and better ones coming in CRDTs, later in this course). You've chosen availability and accepted a merge bill.

State that must never fork. Money, inventory, uniqueness, anything where two histories means calling customers to apologize. This pile gets the full treatment: an odd number of voters, a quorum tiebreak so the minority steps down, and fencing tokens so a zombie's stale authority dies at the storage gate. You've chosen one history and accepted that the small side of a cut goes read-only.

The review questions that expose a design that hasn't decided: "During a partition, which side of this system keeps writing — and what makes the other side stop?" and "When the old primary comes back, what — specifically — rejects its writes?" If the answers are "both" and "nothing," the design has already scheduled its GitHub night; it just hasn't picked the date.

• A partition doesn't take a cluster down; it takes it in two. Both halves stay healthy, each concludes the other is dead, and your own failover machinery crowns a second leader.
• The divergence is the expensive part, not the outage. GitHub: a 43-second cut, then 24 hours and 11 minutes of recovery, because healing the network doesn't merge the histories.
• Quorum is the tiebreak: only one side can hold a majority, so the minority steps down and no second history starts. Keep voter counts odd; two-node pairs need a witness.
• Quorum can't stop the zombie: a paused leader misses the vote entirely and wakes with stale authority on a perfectly healthy network.
• Fencing tokens finish the job: leadership carries a number that only goes up, and the resource itself rejects any write whose number has gone backwards — token 5 bounces off a gate that has seen 6. Quorum stops the second history; fencing stops the old one from coming back.
• Sort your state: may-fork (merge later) versus must-never-fork (quorum + fencing, paid for). A design that hasn't decided has decided.

Every disaster in this lesson began the same way: a timer expired and a detector declared a healthy machine dead. That timer deserves respect — how fast should it fire? What does getting it wrong in each direction cost? Tuning that guess is next: Heartbeats & Failure Detection.