Heartbeats & Failure Detection

This lesson pays a debt. Back in Sync vs Async Replication & Failover, you met the failure-detector dial — "how long do you wait, staring at a leader that has stopped answering, before you declare it dead?" — and the lesson admitted it was ducking the question: "it's a genuinely deep topic and it gets its own treatment later in the course." Later is now.

And the debt has grown teeth since then. The last lesson ended on the observation that every split-brain disaster began the same way: a timer expired, and a detector declared a healthy machine dead. GitHub's 43 seconds became 24 hours through exactly that chain. The frozen node in CAP, Properly was convicted by exactly that chain. The timer is not a config detail. It is the single number where this course's bedrock fact — you cannot tell dead from slow — stops being philosophy and becomes a decision you ship.

In this lesson: how failure detection actually works (it's smaller than you think); the trade that has no correct answer, only a chosen one; why any fixed timeout is wrong somewhere; and the two ideas that make modern detectors respectable — suspicion as a number, and never convicting on one witness.

Scope: this is peers judging peers inside a cluster. The load balancer knocking on backends with GET /healthz was covered in Health Checks & Failover, back in Fundamentals of System Design. How suspicion spreads through a big cluster is Gossip Protocols; what happens after a conviction (the election) is Leader Election. All later in this course. Today: the timer itself.

Strip the mystique off first, because the machinery is almost embarrassingly small.

A node proves it's alive by saying so on a schedule: a tiny "still here" message every interval: a heartbeat. Or the monitor asks and expects an answer within a deadline, a probe (the shape the load balancer used). Either way, the monitor keeps one running judgment per peer: when did I last hear from you?

The verdict rule is one multiplication: tolerate k missed beats at interval T, and your effective timeout is k × T. Heartbeat every second, forgive two misses, convict on the third: a node is "dead" after three seconds of silence. That's the whole detector. Production systems dress it up, but the shipped numbers you met earlier — heartbeat timeouts in the unglamorous 1–5 second range — are exactly this arithmetic.

Theory grades any such detector on two axes, and the vocabulary is worth thirty seconds: completeness (every node that really dies is eventually suspected) and accuracy (no living node gets suspected). Completeness is cheap — silence eventually trips any threshold. Accuracy is the entire game, because the first lesson of this course proved you cannot have it perfectly: silence never tells you which silence this is. Every real detector is a machine for being wrong as rarely, and as cheaply, as possible.

So you must pick k × T. Here is what each direction costs, priced with stories you already own.

Set it tight, say one second, and you buy speed with false accusations. A garbage collector pauses a healthy leader for twelve seconds and your detector convicts it in one. Now the failover playbook runs against a machine that isn't dead: a new leader is promoted, and the old one — remember the zombie — wakes up still wearing its crown. You've read where that goes. A false positive isn't an embarrassing log line; it's an admission ticket to everything Network Partitions & Split Brain just taught you to fear. And ordinary life is full of twelve-second silences: GC pauses, VM migrations, a saturated NIC, an overloaded box.

Set it loose, say sixty seconds, and you buy honesty with downtime. Now no pause ever gets convicted, and when a leader really dies at 3 a.m., your cluster spends a full minute routing writes at a corpse before anyone acts. Your users eat every second of it. The failover lesson called this dial what it is: it sets your recovery time. Patience is not free; it is billed in outage minutes.

There is no third option, because the two costs are pulling on the same rope: every second of patience you add to avoid false positives is a second added to every real recovery. The dial has no correct position. It has a position whose failures you've chosen to prefer.

It gets worse: even if you find a value you like, it's only right for one network, at one time of day.

How Engineers Measure Systems, back in Fundamentals of System Design, made the point that latency is never a number, it's a distribution. Heartbeat arrivals inherit that shape. On a quiet LAN, beats land within a few milliseconds of schedule and a 2-second threshold is enormous. Stretch the same cluster across regions, where round trips are 70–90 ms on a good day and the occasional spike triples that, and the identical 2-second threshold starts convicting healthy nodes every time the network sneezes. Same detector, same number: paranoid in one place, trigger-happy in another, and it drifts with the hour as load changes the tails.

Tuning a fixed threshold by hand, per link, per season, is a losing job. Which raises the obvious question: the detector is already watching every arrival. Why doesn't it learn the distribution itself?

That question has a famous answer, from a 2004 paper by Hayashibara and colleagues: the φ (phi) accrual failure detector. Two moves, both elegant.

Move one: stop answering "alive or dead?" and start answering "how suspicious is this silence?" The detector records the gaps between a node's recent heartbeats and builds the actual distribution — this node, this link, as they really behave. From that it computes a continuous suspicion level, φ, that rises the longer the silence runs: roughly, "given this node's own rhythm, how improbable is a gap this long?" A two-second silence from a metronome-steady peer sends φ soaring; the same two seconds from a jittery cross-region peer barely moves it. The threshold tunes itself to the network's actual behavior — exactly the job you didn't want by hand.

Move two: let different actions demand different certainty. Because φ is a number, not a verdict, the router can stop sending traffic at mild suspicion, while failover — the expensive, zombie-spawning action — waits for near-certainty. One detector, graduated responses.

This isn't exotic. Cassandra's failure detector is this paper, and its conviction dial, phi_convict_threshold, ships at 8 — operators on GC-heavy or cloud-jittery clusters raise it toward 12, consciously trading slower convictions for fewer wrongful ones. Akka ships the same detector for cluster membership. The dial from the failover lesson didn't disappear; it became a threshold on a learned curve, which is a far better place for it to live.

One blind spot remains, and it's sneaky: everything so far assumes the observer is fine. What if the silence is on your end?

Maybe the path between you and the suspect is what broke — the first lesson's whole point was that you can't tell. So before convicting, do what the SWIM family of protocols does: ask someone else to check. "I can't reach node D. Can you three try?" If a random peer gets an answer, D is fine and your route is the problem. One cheap indirect probe converts a false accusation into a network diagnosis. (How this scales into full cluster membership is Gossip Protocols, coming in a few lessons.)

And sometimes the problem is closer still: the judge itself is sick. An overloaded node processes acks late and starts accusing healthy peers; measurements on SWIM found that a single overloaded member is enough to trigger false convictions across a cluster. HashiCorp's Lifeguard fix is almost philosophical: a node that keeps missing acks, or keeps getting refuted, should trust its own judgments less — it automatically slows its own verdicts until it looks healthy again.

Stack the discipline and it mirrors something you already know: suspect individually, confirm collectively, act only on agreement. A conviction that several nodes co-sign is a quorum decision — the same majority machinery that keeps split brain from starting is what keeps detectors from lynching the innocent.

You've been promised this dial for two courses. Time to hold it.

Try this first: run the default hundred seconds and watch the beats land. Then tighten to a 1-second timeout and run again — you'll catch the real crash almost instantly and convict the GC-paused node, twice. Loosen until the false positives vanish and read your new detection lag: half a minute of undetected death. Then flip to the adaptive detector and watch the same stream produce fewer wrongful convictions at the same speed — and notice that even it cannot make the trade disappear. Nothing can. You're not hunting the right setting; you're choosing which failure you'd rather explain at the postmortem.

The discipline that falls out of all this fits on an index card:

Measure before you set. Your heartbeat arrivals have a distribution — look at it (p99, not the average) before picking any threshold. A number chosen without looking is a guess wearing a config key.

Match patience to blast radius. Kubernetes ships its liveness probes patient — checks every ten seconds, three strikes before a restart, roughly half a minute of grace, because a wrongful restart usually costs more than a slow one. Your cheap actions (stop routing to a suspect) can be hair-triggered; your expensive ones (failover, promotion) should demand high φ or a quorum's co-signature.

Suspect fast, convict slow, act reversibly. Route around at first suspicion; fail over on confirmed conviction; and make the wrong call cheap — idempotent retries so a wrongly-suspected node's requests are safe to replay, and fencing tokens so a wrongly-convicted leader can't corrupt anything when it turns out to be alive. You met both tools already; this is the lesson that explains why they're not optional.

One review question exposes an untuned system: "What's your failure-detection timeout, and what did you measure to choose it?" If the answer is a shrug and a default, the timer is still making the bet — just without anyone's consent.

• The whole detector is k × T: heartbeats at interval T, convict after k misses. The 1–5 second production range is this arithmetic, not magic.
• The dial is a bet with a loser on each side: tight → false positives → needless failovers → the split-brain factory; loose → every real death runs undetected → downtime. Every second of added patience is a second added to every real recovery.
• Any fixed number is wrong somewhere: heartbeat arrivals are a distribution, and it changes per link and per hour.
• Modern detectors output suspicion, not verdicts: phi-accrual learns each node's rhythm and asks "how improbable is this silence?" — Cassandra ships it with a conviction threshold of 8, raised on jittery clusters.
• Never convict on one witness: indirect probes rule out your own broken route; Lifeguard's sick-judge insight makes flaky observers doubt themselves; expensive actions wait for quorum agreement.
• Suspect fast, convict slow, act reversibly — and keep idempotency and fencing armed for the day the guess is wrong anyway.

One assumption survived this whole lesson unexamined: that a node which answers its heartbeat is fine. But a heartbeat only proves the heartbeat thread is alive. The disk can be dying, the queries can be timing out, and the little "still here" message still arrives every second with perfect manners. The failures that pass every health check are the worst ones in distributed systems, and they're next: Gray Failures.