Failure Models: Crash, Omission, Timing

The last lesson ended with a loose thread. The eight fallacies tell you what you wrongly assumed about the network. They say nothing about how a node actually fails when an assumption gives way: does it stop cleanly, lie, or just go slow?

That question has a precise answer, and it is older and more useful than most engineers realize. Distributed-systems theory sorts everything a faulty node can do to you into a small set of named failure models, and the names come with a warning label: they are nested, like Russian dolls, and the cost of surviving them climbs steeply as you go up.

Here is the claim this lesson will earn: crash is the kindest thing a node can do to you. Most people's instincts say a crashed server is the disaster. The theory, and every production war story in this lesson, says the opposite. The failures that should keep you up at night are the quiet ones and the chatty ones. By the end you'll rank them by dread, on sight.

In this lesson: the four failure models, each with its real production face; the nesting that turns the list into a price list; and the question that exposes hand-waving in any design review — "we handle failures" ... which class?

Scope: naming the enemy, not fighting it. Detection (heartbeats, timeouts) is two lessons ahead; partitions are next; the unsettling failures that look healthy from one side and broken from another get their own lesson soon (Gray Failures); and the machinery for the top rung gets a one-paragraph map here and its full treatment in ZooKeeper & etcd in Practice, later in this course.

Sort every node failure by one question: how honestly does the node tell you it's broken?

At the bottom of the ladder, the node stops entirely. Painful, but honest. One rung up, it drops some messages and delivers others, and says nothing about which. Another rung up, it does the right work at the wrong time. At the top, it keeps talking, on schedule, in perfect protocol form, and what it says is wrong.

Theory calls these crash, omission, timing, and Byzantine failures, and the ladder is not just a ranking. Each class contains the ones below it. A crash is an extreme omission (the node omits everything, forever). An omission is an extreme timing failure (the message arrives infinitely late). And a Byzantine node can do anything at all, including imitating every failure below. That containment is the whole reason the taxonomy earns its keep: a system that survives rung N survives every rung beneath it, and each rung up demands strictly more machinery. The ladder is a price list.

A crash failure is a node that halts and never acts again. The OOM killer ends your process mid-request. A kernel panics. A power supply dies. From that moment the node sends nothing, receives nothing, corrupts nothing. It has, in effect, resigned.

Why call that kind? Because a crashed node cannot mislead you. It will never acknowledge a write it didn't make, never serve a stale answer, never gossip a corrupted state to its neighbors. Everything it ever told you before the crash was true. The damage has a clean boundary.

Two refinements matter in practice. First, the textbook's politest version, fail-stop, assumes the node halts and every other node can reliably detect that it halted. Read that twice, because it's the assumption real networks refuse to grant: as Welcome to Hard Mode showed with the frozen node that got convicted as dead, you cannot detect a remote death, only suspect it after a timeout. Fail-stop is lovely to write proofs about and does not exist on the open internet.

Second, real machines rarely stay dead. The crash-recovery model adds the realistic wrinkle: the node reboots ninety seconds later and rejoins, having forgotten everything it hadn't written to disk. That amnesia is precisely why databases keep write-ahead logs — WAL & Durability: Why Committed Data Survives, back in Data Systems at Scale, is the mechanism that lets a crashed node come back as a citizen instead of a hazard.

The machinery bill for crash tolerance is the cheapest on the ladder: keep a copy of the state somewhere else, restart the process, catch it up. Replication plus a supervisor. This is the rung the entire mainstream stack is engineered around, and the rest of the lesson explains why that's both sensible and quietly dangerous.

One rung up, the node stays alive but some of its messages simply don't happen. A request reaches it and no reply comes back, or the reply is sent and never arrives. Theory splits hairs between send-omission and receive-omission; the operational reality is the same either way: silence, sometimes.

You already know omission's face because the first lesson measured it: tens of links and several devices failing daily inside a single well-run fleet, tens of thousands of packets lost per failure. A switch queue fills and drops your packet without a note of apology. A NIC hiccups. An overloaded server accepts a connection and never gets around to answering it. Omission is the network's default misbehavior, which is why the fallacy it punctures sits at the top of the list.

What makes omission nastier than crash is the inconsistency. A crashed node fails everyone, always — the picture is at least coherent. An omitting node works for your colleague and fails for you, passes the health check and drops the real request, works at 09:00 and not at 09:02. Debugging it feels like interrogating a witness who answers every third question.

The machinery bill: retries, because a lost message can be resent, plus idempotency so the resend is safe when the first attempt secretly succeeded and only its reply was lost. You built both muscles in Fundamentals of System Design; the retry side gets its deep treatment in Resilience & Overload Control, later in this course.

The third rung produces correct work at an unacceptable time. A timing failure is the node that answers in thirty seconds when the contract was thirty milliseconds: a garbage-collection pause freezes the process mid-request, an overloaded box queues your call behind a thousand others, a clock drifts and fires a task early.

Timing failures only exist once you expect bounds. On a LAN with a deadline, a slow reply is a failure; in a batch job with no deadline, the same delay is just Tuesday. That's why theorists tie this class to assumptions about synchrony, and why practitioners meet it as the most ambiguous thing on the pager.

And here the ladder connects back to the bedrock fact of this course. You saw a healthy node get frozen (a fake GC pause), stay silent past the timeout, and get convicted as dead by its cluster while sitting there in perfect health. That is a timing failure being indistinguishable from a crash. From the observer's side there is no experiment that separates "dead" from "slow" from "my request was dropped" — three different rungs of the ladder, one identical observation: silence past the deadline.

The machinery bill: timeouts and the humility to treat their verdicts as guesses. You stop waiting at some point you chose, you act on suspicion rather than knowledge, and you engineer so that a wrong guess (retrying against a node that was merely slow) is safe rather than fatal. The tuning of that guess — heartbeat intervals, timeout values, false-positive trade-offs — is two lessons ahead, in Heartbeats & Failure Detection.

At the top of the ladder the node does the one thing the other three never do. It keeps talking, on time, in perfectly valid protocol form, and what it says is wrong.

Theory calls this a Byzantine (or arbitrary) failure, after a 1982 paper by Leslie Lamport, Robert Shostak, and Marshall Pease that dressed the problem as generals coordinating an attack while some of them are traitors. The name suggests malice, and that's the first thing to unlearn: in production, Byzantine behavior is overwhelmingly bugs and hardware. A bit flips in RAM and a counter is suddenly enormous. A disk's firmware acknowledges a write it silently dropped. A bad release computes a plausible-but-wrong balance. No attacker required.

The canonical war story is painfully simple. On July 20, 2008, one Amazon S3 server sent a state message with a single flipped bit. Customer data was checksummed; these internal gossip messages were not, so nothing caught the corruption, and the gossip protocol did exactly its job: it spread the poisoned state across the fleet. Servers drowned in state chatter, and S3 was down for roughly eight hours. One bit. No crash, no attacker: a healthy-looking machine confidently repeating a falsehood, and a system built to trust whatever a peer said.

Why is this rung priced so differently? Because every defense below it assumes that what arrives intact is true. Retries re-fetch the same lie. Timeouts don't fire, since the liar is punctual. The only cure theory offers is redundancy in a stronger form: don't trust any single answer. Compare copies, sign messages, take votes. The 1982 result puts a hard number on it: with plain messages, tolerating f liars requires 3f + 1 nodes — three generals cannot survive one traitor. That number is why full Byzantine fault tolerance stays rare and expensive, and we'll map where it actually lives (coordination services assume honest nodes; public blockchains do not) in ZooKeeper & etcd in Practice.

You now have four names. What the names don't give you yet is the feel of why the price climbs — why a defense that handles the bottom three rungs waves the top one straight through.

Try this first: make node B crash, fire a few requests, and arm retry-plus-timeout. Contained. Now make it lie instead, leave your defenses exactly as they are, and fire again. Watch the well-formed wrong answer sail through retry, through timeout, through the checksum, and land in your client as truth. Then arm the vote and see what catching it actually costs.

Line the rungs up with their bills and a pattern appears:

So here's the honest question a sharp reviewer will ask: if Byzantine failures are real — and S3's flipped bit says they are — why does almost nothing you use run Byzantine fault tolerance?

Three reasons, in increasing order of interest. Cost: 3f + 1 replicas plus heavier voting rounds, versus a leader and a couple of followers for crash tolerance. Trust: inside one company's datacenter, nodes aren't adversaries; the realistic enemies are bugs and hardware, not traitor generals. And third, the interesting one — demotion. Engineers don't usually tolerate Byzantine failures; they demote them down the ladder. A checksum turns a corrupted message into a detectably corrupted message, which you drop — and a dropped message is an omission, which retries already handle. TLS and authentication turn a forged message into a rejected one. ECC memory turns a flipped bit into a corrected bit. That is exactly what the S3 team did the week after the outage: they didn't deploy Byzantine consensus, they added checksums to the gossip — armor that shoves the top rung's failures down to a rung they could already afford. It's also why Checksums: Trust but Verify, back in Fundamentals of System Design, was never really a lesson about networking trivia.

So the industry's standard bet is: build for crash-recovery plus omission, guess carefully at timing, and wear demotion armor against the top rung. That bet is correct for almost everyone. It fails in two places: systems whose participants genuinely can't trust each other (public blockchains take the expensive route on purpose), and — far more common — systems that wear the armor incompletely. S3 checksummed customer data and forgot its own gossip. The lesson of the ladder isn't "pay for the top rung." It's: know which rung you've built to, on purpose, and where your armor has gaps. "We handle failures" is not a sentence a reviewer should let you finish. Which class?

• Four models, ranked by politeness: crash (halts honestly) → omission (silence, sometimes) → timing (right answer, late) → Byzantine (confident lies). Dread goes UP the ladder; crash is the kindest failure a node can offer you.
• The ladder nests. Each class contains the ones below, so surviving rung N covers everything beneath — and costs strictly more: restart < retry < timeout < 3f + 1 voting.
• Real faces: the OOM kill (crash), the fleet's daily lost packets (omission), the frozen node convicted as dead (timing), S3's single flipped bit spreading through unchecksummed gossip for eight hours (Byzantine — bugs and hardware, not spies).
• The trick is demotion, not tolerance: checksums, auth, and ECC shove Byzantine failures down to droppable garbage, which is why crash-recovery + retries + armor is the standard, sane bet — when the armor has no gaps.
• The review question: "we handle failures" — which class? Name the rung you defend to and the failures you've chosen not to survive.

One failure got conspicuously left out. Everything here was a single node misbehaving. Next, the failure that isn't a node at all: the network splits, both halves are perfectly healthy, and each concludes the other is dead — Network Partitions & Split Brain.