Welcome to Hard Mode

For two whole courses, we kept making you a promise we wouldn't keep.

Every time the hard part showed up, we flinched. Replication Topologies told you that duplicating state safely needs "a single writer or consensus — or you get split-brain," and then didn't tell you what consensus was. Sync vs Async Replication & Failover walked you right up to the moment the old leader dies and the survivors have to pick a new one — and stopped, calling that agreement "the hardest thing in distributed systems" and leaving it there. Version Conflicts showed you two writes that will fight forever, then pointed straight here — "that's a CRDT, and we build them properly" in the course you're now starting.

Welcome to Distributed Systems & Reliability. This is where every one of those IOUs comes due.

In this lesson: why a network of computers is not just a bigger computer but a categorically different thing to reason about; the single sentence this whole course is a response to; the one rule that tells this course apart from the next; and the map of the seven sections ahead.

Scope: this is the map, not the territory. Every mechanism named here — fallacies, failure models, partitions, clocks, consensus, CRDTs, circuit breakers, SLOs — gets its own full lesson. Today we only earn the right to be afraid of the right things.

Here is the thing nobody told you while you were learning to program: a single computer is honest.

It runs your instructions in order. If you write a value and read it back, you get the value. If you call a function, it either returns or the whole process crashes — and if it crashes, the operating system knows, and it tells you. Failure on one machine is total and it is detectable: the program is either running or it is not, and there is a single authority — the OS — that can see the whole thing and adjudicate.

This is what Kleppmann means when he says a single computer is deterministic: give it the same inputs and it produces the same result, and if an operation fails you can simply try it again and get the same answer. There is no "maybe." The reason you never had to think about any of this is that it was true, reliably, for free, the entire time you had one box.

Then you added a second box, and it stopped being true.

The moment two machines have to talk over a network to get one thing done, you inherit a new substrate underneath your program — and that substrate lies.

A message you send can vanish and never arrive. It can arrive, but late — so late you'd given up. It can arrive twice. It can arrive out of order, so the second thing you said lands before the first. The clock on the other machine disagrees with yours about what time it is. And the machine itself can be dead, or alive-but-frozen, or perfectly fine but unreachable — and from where you're standing, those look identical.

In 1994, a team at Sun led by Jim Waldo wrote a paper — A Note on Distributed Computing — whose entire purpose was to kill the seductive idea that a remote call is just a slower local call. It named four things that turn intrinsically harder the moment a call crosses a network — latency, memory access, concurrency, and the one that swallows the rest for us here: partial failure. On one machine, failure is total: the whole process dies together and the OS sees it. Across machines, failure is partial, and here is the sentence to carry: there is no common agent. No single observer sees the whole system, decides what failed, and tells the survivors. One component lies dead while another sails on, cheerfully unaware. Any system that tries to hide this difference, they argued, "fail[s] to support basic requirements of robustness and reliability."

Leslie Lamport had already put it more cruelly, in a 1987 email that became the most-quoted sentence in the field:

"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable."

In 1987 that was a wry thought experiment. Today it's your architecture diagram. A single tap in a modern app fans out to hundreds of backend services owned by teams you've never met — Netflix went from 500-odd microservices in 2012 to thousands now, and had to build a system whose only job is to keep a live map of them because no human can hold the graph in their head. Your checkout's correctness depends on a computer you not only didn't know existed — you couldn't have.

And this is where people wave it away with "sure, but the network is basically reliable." And here's the honest part: it mostly is — Microsoft's own study of their datacenter fleet concludes these networks are highly reliable. But "highly reliable" is not "reliable," and the gap is where you live. Averaged across that fleet, every single day:

Links and devices failing every day, in a fleet the authors themselves call highly reliable — and that's before you cross the open internet, where a five-year study of one research network logged 508 isolating partitions, some lasting days. The redundancy you bought is not a magic shield: PagerDuty once went fully dark for 18 minutes because two "independent" datacenters quietly shared a single peering router — a computer nobody knew was a single point of failure until it wasn't.

So: not a single machine with more problems. A different kind of thing.

Of all the ways the network lies, one is load-bearing for this entire course, so slow down here.

You sent a request to another node and no reply came back. Why? Maybe the node crashed. Maybe it's alive but paused in a 30-second garbage-collection freeze. Maybe it answered instantly and the reply was the packet that got lost. Maybe the cable is cut. From your side, every one of those produces the exact same evidence: silence. You cannot distinguish a dead node from a slow one, because the network gives you no signal that tells them apart.

You already met this in CAP, Properly — the frozen Cassandra node that the cluster convicted as "down" while it sat there in perfect health. What we do about the silence is guess: we start a timer, and when it runs out we conclude the node is gone. That conclusion is not a detection. It's a bet.

And that single undecidable fact is the seed of this whole course. Because no node can be sure whether it is the one that got cut off, no node can trust its own judgment about the state of the world. So "truth" in a distributed system can't be what any one machine believes. From there you have exactly two escapes, and this course teaches both. If you need one authoritative, agreed-upon answer even through a partition, it has to be blessed by a majority — that's not a design choice, it's forced, and it's exactly why consensus has to exist (we get there in Consensus & Coordination). Or, if you can let the copies disagree for a while and merge them later, you sidestep the vote entirely — the leaderless escape hatch that Collaborative & Replicated State is built on. What's off the table is the naive move: trusting your own view. Hold the thread that gets you to both: you can't tell dead from slow → you must guess → so you can't just trust yourself. Sections 2, 3, and 4 are three answers to "how do you agree when the substrate lies?"

Reading that a network is nondeterministic is not the same as feeling it, so here is the same operation run in two worlds.

Try this first: run the request against the single machine a few times — it's boringly honest, same answer every time. Now flip to the distributed world and fire the identical request: turn on the lies (drop, delay, duplicate, reorder) and watch the network misbehave. Then one node comes back silent — and you have to make the call the machine has to make: is it dead, or just slow? Commit, and watch the widget reveal that you were guessing. The point isn't that things fail — it's that you can no longer tell what failed, or whether it failed at all.

One piece of housekeeping that will save you real confusion later, because the next course looks deceptively similar to this one.

Every production system eventually forces two different questions, and they belong to two different courses:

"How do I not fall over when something breaks?" — that's this course. Surviving failure. It's why multi-region, disaster recovery, and SLOs live here: they are all availability engineering, the discipline of staying up when parts of you die.

"How do I run, watch, secure, and pay for this thing, day after day?" — that's the next course (Architecting Production Systems). Running and evolving: deployments, observability, security, cost.

When you're unsure which course a topic belongs to, ask which question it answers. Keeping a region alive through an outage is surviving failure (here). Rolling out v2 without downtime is running and evolving (there). Same systems, two different jobs.

Here is the whole course on one page, and it is deliberately a journey, not a pile of topics — each section hands the baton to the next.

1. Why Distribution Is Hard — the diagnosis you're reading now: what the network really does to you.
2. Time & Order — it gets worse: the machines can't even agree what time it is, or which of two events happened first.
3. Consensus & Coordination — so how does a group of liars agree on one value anyway? This is the IOU — leader election, Raft, the machinery that stops two nodes from both believing they're in charge.
4. Collaborative & Replicated State — and how do you agree when there's no leader at all — two people typing in the same doc, offline, on a train (CRDTs)?
5. Resilience & Overload Control — now stay up: timeouts, retries, circuit breakers, backpressure, and the tail latency that eats you at scale.
6. The Reliability Discipline — make reliability a number you budget (SLOs and error budgets) and a practice you run (incidents, disaster recovery, multi-region).
7. Learning from Real Outages — the bill, itemized: Facebook's six-hour BGP self-lockout, the AWS cascades — then you make your marketplace survive all of it in Capstone Pass 3.

Notice the arc: sections 1–2 are the problem, 3–4 are agreeing anyway, 5–6 are staying up and managing it, and 7 is the reckoning. You descend into how bad it is, then climb back out with the tools.

Now the part most courses skip, because it undercuts the drama: you should avoid all of this whenever you possibly can.

Distribution is not a virtue or a maturity badge. It is a tax you pay to buy exactly two things — scale past what one machine can hold and availability past what one machine can guarantee — and the tax is everything this course spends the next 47 lessons clawing back: correctness, agreement, the ability to debug at 3 AM. So the senior first move on any design isn't "how do I distribute this?" It's the opposite question: "do I have to be in hard mode at all?" A single well-chosen box keeps every guarantee you just learned to miss — no partition, no consensus, no dead-or-slow — for free, and it is the correct answer for a huge number of real systems. (A primary with a warm standby buys durability cheaply too — but watch the fine print: the moment its failover becomes automatic, you have quietly re-entered hard mode, because now something has to decide the primary is truly dead and not just slow, and do it without promoting two. That is this whole course's problem, which is why "just add a standby" is rarely as free as the diagram makes it look.)

This isn't a beginner's cop-out; it's what the best teams actually do. Martin Fowler's rule of thumb after watching the industry for a decade: "Almost all the successful microservice stories have started with a monolith that got too big and was broken up. Almost all the cases where I've heard of a system that was built as a microservice system from scratch, it has ended up in serious trouble." And in 2023, Amazon — the company that invented the two-pizza-team microservice — published a case study on its own Prime Video quality-monitoring service, which had been built the textbook distributed way with Step Functions and S3 between every stage. It hit a wall at 5% of the target load and cost too much to run. The fix that made the front page: they collapsed it back into a single process, passed the data in memory instead of over the network, and cut costs ~90%.

Read that correctly — it's not "monoliths win." It's "distribution is a trade-off, and for this workload they'd taken the wrong side of it." Which is the whole point. Enter hard mode on purpose, late, and only when the ceiling or the availability target is genuinely real. When it is — when you truly can't fit on one machine or can't afford to fall when one machine does — then everything in this course is the toolkit that keeps you correct anyway. That's when the tax is worth paying.

• A single machine is honest; a network of machines lies. One box is deterministic — up or down, and it tells you. Add a network and messages vanish, delay, duplicate, and reorder; clocks disagree; nodes are dead, slow, or unreachable and you can't tell which.
• Partial failure is the categorical difference. On one machine failure is total and the OS sees it; across machines it's partial and there is no common agent that sees the whole and decides what died (Waldo, 1994).
• Dead is indistinguishable from slow. You can't detect it, you can only guess with a timeout — which is why no node can trust its own judgment, and truth must be agreed by a majority. That single fact forces consensus.
• This course = surviving failure; the next = running & evolving. Ask which question a topic answers to know where it lives.
• The arc: why it's hard → agreeing anyway → staying up → the reckoning (Capstone Pass 3).

Next — The Fallacies of Distributed Computing: we said the network "lies" in general; now we name the eight specific lies you'll tell yourself, starting with the mother of them all — the network is reliable.