Health Checks & Failover

For four lessons we've handed the load balancer more and more power. We gave it a front door, taught it to choose a server (round-robin, least-connections), and decided what it's allowed to read (L4 vs L7). But every one of those decisions quietly assumed something we never checked: that the servers it's choosing among are actually alive.

They aren't, not always. Boxes crash. Processes deadlock. A deploy leaves one replica half-booted. A disk fills. And here's the sharp part — routing a request to a dead server is worse than having no load balancer at all. With no balancer, the user hits a live box and gets an answer. With a balancer that doesn't know a box is dead, the user gets routed straight into a black hole and sits there until something times out. The balancer, trying to help, made the failure slower and more confusing.

So a real load balancer never trusts its full list of servers. It keeps a smaller, living list — a roster of backends it currently believes are healthy — and it only ever routes to that. This lesson is about the machinery that keeps that roster honest: health checks (how the balancer senses which boxes can still take work) and failover (what it does the instant the roster changes). It sounds like plumbing. It contains two of the most spectacular self-inflicted outages in all of systems — and the fixes that stop them.

The simplest way to know if a server is alive is to ask it, on a schedule. That's an active health check: the load balancer reaches out to each backend every few seconds and knocks on the door. The knock can be as shallow as a TCP connection ("did anything answer the port?") or a real HTTP request — GET /healthz, expecting a 200 OK back within a timeout.

The numbers are worth knowing because they are the behavior. On an AWS Application Load Balancer the defaults are an interval of 30 seconds and a timeout of 5 seconds — every half-minute the balancer pings each target, and if nothing comes back within five seconds, that ping counts as a failure. nginx Plus and F5 do the same continuously: probe the upstreams, steer around the ones that fail, and quietly slide a recovered box back into rotation once it starts answering again.

The strength of active checking is that it works with zero user traffic — a server can crash at 3am with nobody online and the balancer still notices within a cycle. The weakness is that it only knows what it thought to ask. A box can answer GET /healthz with a cheerful 200 while every real request to it returns a 500, because /healthz returns a hardcoded "OK" and never touches the code path that's actually broken. The probe passes; the users suffer. Which is exactly why balancers also watch.

The other way to know a server is sick is to watch how it handles real traffic — no separate probe at all. This is a passive health check: the balancer monitors the actual requests flowing through it, and when a particular backend starts racking up connection failures, timeouts, or 5xx responses, it marks that box down and stops sending it work until it looks healthy again. nginx phrases it plainly — it monitors transactions as they happen and, when one can't be resumed, pulls the server out until it recovers. Envoy gives it a fancier name, outlier detection: automatically ejecting the host that's "performing unlike the others."

Passive checking is cheaper — no synthetic probe traffic — and it catches the failure the probe missed: the box that answers /healthz fine but 500s real requests can't hide from the requests themselves. Its weakness is the mirror image of active checking's strength: it only learns a box is bad after real users have already hit it. The first few unlucky requests are the sensor.

So production systems don't choose. They run both: active checks to catch a box that died while idle, passive checks to catch a box that lies to the probe but fails the users. Active asks; passive watches; together they see far more than either alone.

Here's a mistake that looks like caution and behaves like chaos: ejecting a server the instant one check fails. Servers hiccup constantly for reasons that have nothing to do with being dead — a garbage-collection pause freezes it for 200ms, a single packet drops, the box is mid-way through a big request. If one failed probe meant "dead," your healthy fleet would be a strobe light.

So every real balancer uses thresholds with hysteresis — a deliberate reluctance to change its mind:

• Eject a box only after N consecutive failures (UnhealthyThresholdCount).
• Re-admit it only after M consecutive successes (HealthyThresholdCount).

That asymmetric stubbornness is what prevents flapping — a box bouncing in and out of rotation every few seconds, which churns connections and often hurts more than just leaving it out. And it hands you the single most important dial in this whole lesson: detection time ≈ interval × unhealthy-threshold. Short interval and a low threshold catch death fast but also eject boxes that were merely slow; long interval and a high threshold are rock-steady but leave users hitting a dead box for longer. There's no setting that's both instant and never-wrong — you're choosing where on that line to sit, and the right answer depends on whether your "failures" are real death or transient slowness. (Envoy adds a nice touch: a repeat offender gets ejected for longer each time — base_ejection_time × consecutive ejections — so a chronically flaky box stops getting quick second chances.)

This is the lesson you have to watch, because a roster changing in real time is the whole point and a paragraph can't move. Below is a live load balancer probing a pool of backends and sending real traffic only to the healthy ones. You break it.

Start by clicking a server to sick it (it begins erroring) or kill it (hard down). Watch its health probes turn from green to red, watch the failures pile up, and — this is the beat the whole lesson turns on — watch it flip to ejected the moment it crosses the unhealthy-threshold, with the traffic rerouting to the survivors in real time. Click it again to recover it and watch it count clean probes until it earns its way back in.

Then spring the two traps that make this topic famous. Flip the check to deep and press blip the database: a check that asks "is the DB up?" turns every box red at once, and without a safety net the balancer ejects the entire fleet over a three-second wobble. Push more than half the pool unhealthy and you'll trip panic mode — the balancer gives up on its own roster and routes to everyone, because a maybe-degraded box beats an empty pool. And sick a couple of boxes with retries on and no budget to watch the retry storm spiral. Every one of those is a section below — but you'll understand them faster by breaking them here first.

"Is this server healthy?" is actually two questions wearing one coat, and prying them apart fixes a surprising number of production mysteries.

• Liveness asks: is this process alive, or is it wedged? — stuck in an infinite loop, deadlocked, out of memory. If liveness fails, the right response is to restart the process. It's beyond saving; reboot it.
• Readiness asks: can this process serve a request right now? If readiness fails, the right response is to pull it out of the load balancer's rotation — but not kill it. Nothing's wrong with it; it's just not ready.

The gap between them is real and common. A box that just started is alive but not ready — it's warming a cache, opening connection pools, loading a model. A box at capacity is alive but shouldn't get more work this second. If you route to it (treating alive as ready), users hit a cold, struggling server. If you restart it (treating not-ready as not-alive), you murder a box that needed five more seconds to warm up — and under a traffic spike, you restart the whole fleet right when you need it most.

That's why the guidance is blunt: keep liveness and readiness separate and independent. One decides reboot or not; the other decides route or not. They are not the same signal, and wiring them together is how a warm-up delay becomes a restart loop.

Now the counterintuitive heart of the whole lesson — the mistake that feels like diligence and detonates like a bomb.

You write a health check. You think: a shallow "is the process up?" is weak — a server could be up but unable to reach the database, and then it's useless. So you make the check smarter. /healthz now also opens the database, runs a SELECT 1, checks the cache, pings a downstream service. Now a green check really means something. Feels obviously better.

It is a fleet-wide time bomb. Because you just made every backend's health depend on the same shared thing. Watch what happens when the database has a routine three-second blip — a failover, a slow query, a brief network hiccup. Every backend's deep check tries to reach the DB. Every one of them times out. Every one reports unhealthy at the exact same instant. The load balancer looks at its roster, sees zero healthy servers, and ejects the entire fleet. A minor, self-recovering dependency wobble just became a total, hard outage — and the load balancer, following your instructions perfectly, is the thing that caused it. AWS says it in a sentence: a slow dependency check can cause every readiness probe to time out, removing every instance from the load balancer.

The rule that falls out of this is one of the most important in reliability engineering, and it sounds backwards until you've felt it: a liveness/health check must not depend on external or shared dependencies. Ask "can this process serve a request?" — not "is the whole world healthy?" The best health check is almost dumb on purpose. If you genuinely need to know the DB is reachable, make it a separate signal that alerts you — never one that lets the balancer eject boxes. Depth in a health check doesn't add safety; it adds correlation, and correlated failure is the whole enemy.

Even with a shallow check, the balancer needs a backstop for the moment its roster goes to zero — because "every server is unhealthy" almost never means every server actually died. It usually means the health check itself is wrong: a bad deploy of the check, a network partition between the balancer and the pool, a deep check someone slipped in. If the balancer blindly trusts a roster that says "nobody's healthy," it stops routing entirely and guarantees the outage.

So good balancers refuse to believe it. This is panic mode: when more than a threshold fraction of backends are marked unhealthy — Envoy's default is 50% — the balancer concludes the sensor is lying, not that the fleet vanished, and starts routing to everyone again, ejected boxes included. Its companion knob, max_ejection_percent, enforces the same instinct from the other side: never eject below a minimum capacity (keep, say, half the pool in rotation no matter what).

The philosophy is worth saying out loud, because it's the opposite of how health checks feel like they should work: it is better to send traffic to a maybe-degraded server than to send it to a guaranteed-empty pool. Some of those "unhealthy" boxes are probably fine; a degraded response beats a definite failure. Panic mode is the system admitting that its own senses can break — and choosing to try anyway rather than confidently route into the void.

Detecting a bad box is half the job; removing it without collateral damage is the other half. Yank a server out of rotation the instant it fails and you guillotine every request it was mid-way through serving — a user's upload dies at 90%, a checkout half-commits. So removal is done gracefully, with connection draining (AWS calls it the deregistration delay, default 300 seconds): the balancer stops sending the box new requests immediately, but lets the in-flight ones finish before it fully cuts the cord. An ALB target visibly walks through states — Initial → Healthy → Unhealthy → Draining — and Draining is exactly this courtesy.

Two more pieces complete graceful failover:

• Timeouts are the caller's own health check. Draining protects requests on a box that's leaving cleanly; a request timeout protects the caller from a box that's hung right now. Never let one stuck backend hold a caller hostage forever — cap the wait, give up, and try elsewhere. A timeout is how the client independently decides "this one's dead to me."
• The balancer is a single point of failure too. It sits in the request path, so if it dies, the healthiest fleet on earth is unreachable — the exact "the front door is a SPOF" problem from the very first lesson, one level up. So the balancer itself is run redundantly: an active-passive pair sharing a floating virtual IP that jumps to the standby when the primary fails its own health check, or an active-active set. The thing that watches everyone else needs a watcher too.

Here's the second famous self-inflicted outage, and it hides inside a feature that sounds purely good: retries. A request fails on one box, so the balancer transparently retries it on another. The user never notices the blip. Lovely — when the failure is isolated.

Now picture the failure isn't isolated: your fleet is overloaded, not dead — every box is slow and shedding a few percent of requests. Retries turn that from a bad day into a death spiral. Each failed request becomes two or three requests. That extra load lands on the boxes that were already struggling, so they slow down more and fail more, so the passive health checks start ejecting them, so the survivors inherit even more load, so they fail too. The documented Envoy spiral reads like a curse: eject → fewer endpoints → more load → more 5xx → more eject. Your resilience mechanism became the outage. The retries meant to route around failure instead amplified it into total collapse.

The fixes are about capping the amplification, not removing retries:

• Retry budgets — the rule that retries may never exceed a small percentage of total traffic (say 10–20%). When the fleet is broadly failing, the budget is exhausted and retries simply stop, starving the storm of fuel.
• Exponential backoff with jitter — don't retry instantly or in lockstep; wait a little longer each time, randomized, so the retries don't arrive as a synchronized wave.
• The circuit breaker, the caller's version of ejection. It has three states: Closed (normal — requests flow, failures are counted), Open (after too many failures it trips and fails fast, not even calling the sick box, giving it room to recover), and Half-open (after a cooldown it lets a trickle of test requests through — if they succeed it closes, if they fail it re-opens). Where the balancer ejects a box it's watching, the circuit breaker lets the caller stop hammering a dependency that's clearly down. (We'll go deep on it in resilience patterns later; know the shape now.)

Strip the features away and health-checking is a handful of dials, each of which buys one thing by spending another. Knowing the price is what separates a config you copied from a config you chose:

• Interval × threshold — speed vs false positives. Tighter settings catch a real death in seconds but start ejecting boxes that were only slow; looser settings are stable but leave users on a dead box longer. There's no "instant and never wrong." Measure whether your failures are actual death or transient slowness, and set the dial to match.
• Deep vs shallow — coverage vs correlation. A deeper check catches more genuine problems (the up-but-broken box) but correlates failures across the whole fleet, which is how one blip ejects everyone. A shallow check can't catch a broken-but-up box but can't take down the fleet either. Default shallow; add depth only as a non-ejecting alert.
• Retry aggressiveness — masking vs amplifying. More retries paper over more transient blips, right up until they turn a partial failure into a total one. Budget them.

And the trade-off underneath all three, the one to actually internalize: health checking makes a system self-ejecting, not automatically self-healing. It's a feedback loop with the power to remove capacity — and a feedback loop that removes capacity under stress can remove all of it. Panic mode, max_ejection_percent, and retry budgets aren't optional garnish; they're the governors that keep self-ejecting from becoming self-destructing.

Four beliefs that sound responsible and quietly cause outages:

• "A deeper, smarter health check is a better one." Backwards. Depth correlates failures — one shared-dependency blip fails every box's check at once and ejects the fleet. The best liveness check is almost deliberately dumb: can this process serve a request?, nothing more.
• "Health checks make my system self-healing." They make it self-ejecting. Sensing a bad box and pulling it is only safe when something stops the pulling from cascading — panic mode, a max-ejection cap, retry budgets. Without those, "self-healing" is one dependency blip away from self-destructing.
• "If a box fails a health check, it's dead — pull it immediately." Maybe it's mid-GC, or warming a cold cache, or finishing a big request. That's why thresholds exist (don't trust one blip) and draining exists (don't kill work that's in flight). Instant, single-signal ejection is how you strobe a healthy fleet.
• "Retries make me more resilient." Unbudgeted retries turn a partial failure into a total one — the storm. Real resilience is retries with a budget, backoff, and jitter, plus a circuit breaker so the caller stops hammering a box that's clearly down.

Twenty minutes and you'll have watched every idea in this lesson happen on real processes — and, more valuably, watched the traps happen so you recognize them in an incident.

1. Stand up three tiny servers behind nginx or HAProxy, each exposing a /healthz that returns 200 only when the process itself is ready — crucially, without touching any database. (Keep the check shallow on purpose; you're about to see why.)
2. Turn on active health checks (nginx Plus health_check, or HAProxy option httpchk with check inter 2s fall 3 rise 2). Send a steady trickle of traffic with hey or wrk, then freeze one worker with kill -STOP <pid>. Watch the balancer eject it after the configured consecutive failures, watch traffic reroute to the other two, then kill -CONT it and watch it earn its way back after the rise count. That's the core loop, live.
3. Now build the deep-check bomb — safely. Add a second endpoint /healthz-deep that also opens the database, and point the balancer at it. Kill the database (or block its port with a firewall rule) for ten seconds. Watch all three servers go unhealthy at once and the balancer eject the whole fleet from a single dependency blip. Restore the DB, switch the balancer back to the shallow /healthz, repeat the blip — the servers stay in rotation. You just reproduced a real class of outage in your terminal.
4. Feel the retry storm. Add a client-side retry with no budget, overload the fleet slightly, and watch p99 and error rate climb as retries pile on. Then add a retry budget (or a circuit breaker like resilience4j / Polly / Envoy's retry_budget) and watch the spiral flatten.

The lesson isn't the config syntax — it's the muscle memory of inducing the failure and watching the balancer's reaction, especially the reactions that make things worse.

A load balancer never trusts its raw list of servers — it trusts a live roster of healthy ones, and this lesson was the machinery that keeps that roster honest:

• Active checks ask each box on a schedule; passive checks (outlier detection) watch real traffic. Run both — one catches idle death, the other catches boxes that lie to the probe.
• Thresholds + hysteresis stop a single blip from ejecting a live box; detection time ≈ interval × threshold is the dial between fast and false-positive.
• Liveness ≠ readiness: one decides restart, the other decides route — keep them separate.
• The deep-check trap: a check that depends on a shared dependency correlates failure and ejects the whole fleet on one blip. Keep the check shallow; check this process, not the world.
• Panic mode + max-ejection are the backstop: >50% unhealthy means the sensor is probably wrong, so route to everyone rather than an empty pool.
• Graceful failover = connection draining + timeouts, and the balancer itself runs redundantly (it's a SPOF too).
• The retry storm: unbudgeted retries amplify a partial failure into total collapse — cap them with budgets, backoff, jitter, and circuit breakers.

The thread: health checking makes a system self-ejecting, and the entire craft is keeping self-ejecting from becoming self-destructing.

We've now built a load balancer that spreads load intelligently, routes by content, and quietly routes around death — for one pool, in one datacenter. But your users aren't in one datacenter. They're in Mumbai and São Paulo and Frankfurt, and the speed of light doesn't care how healthy your Virginia fleet is. Balancing across continents — sending each user to the nearest healthy region — is a different game with different tools: GeoDNS and Anycast, the next lesson.