Circuit Breakers

The last two lessons gave you tools that assume the dependency will eventually answer: a timeout to bound the wait, a budget to bound the retries. Both are still calling the service. This lesson is for the case they can't handle: when the service isn't slow, it's gone, and every polite, bounded, well-jittered call is a thread thrown into a hole.

The tool is the circuit breaker, named by Michael Nygard in his 2007 book Release It! after the thing in your wall that cuts the current before the wiring melts. You met the three-state sketch of it in Calling Services Safely: Timeouts, Retries & Backoff. This lesson is the full machine: how it decides a dependency is dead, how it tests whether it's back without re-killing it, and where it lives. And underneath, one idea you already know well, because a breaker is a failure detector on the call path, and every hard question about it is the same bet you first met with heartbeats: trip too eagerly and you cut off a service that was fine; trip too slowly and you keep feeding a corpse.

The breaker is a small state machine wrapped around every call to one dependency. Three states, and the whole design is in the arrows between them.

Closed is normal and invisible. Calls flow straight through, and the breaker does one quiet job: it watches, tallying successes and failures. As long as the dependency is mostly healthy, it stays closed forever.

Open is tripped. Once failures cross a threshold, the breaker flips open and stops calling the dependency at all: every request returns immediately with an error or a fallback, without a socket, without a timeout, without a held thread. This is the state that saves you: the load on the dead service drops to nothing, which is often exactly what it needs to recover. A cooldown timer runs while it's open.

Half-open is the careful test. When the cooldown expires, the breaker doesn't fling the doors back open — it lets a small, counted number of calls through as probes. If they succeed, the dependency is back: close the breaker, resume normal traffic. If they fail, it isn't: snap straight back to open and wait again.

Watch, stop, test, decide. That loop is the entire pattern, and every knob in a real breaker library is just a number attached to one of those arrows.

"Once failures cross a threshold" hides every hard decision in the pattern. Get the threshold wrong and the breaker either never helps or becomes the outage itself. Three things make it work.

A rate, not a count — over a window. "Ten failures" is meaningless without "out of how many." Real breakers trip on a failure rate measured over a sliding window: the last hundred calls, or the last ten seconds. resilience4j's default is 50% over a window of 100 calls; Netflix's Hystrix used 50% over a rolling ten seconds. Below the line, closed; above it, open. Some breakers count consecutive failures instead — a service mesh like Envoy ejects a host after five 5xx in a row — which reacts faster but is noisier.

A floor, so noise can't trip it. A single failed call in an otherwise-quiet minute is not an outage, and a breaker that trips on it flaps uselessly. So every real breaker has a minimum number of calls before the rate even counts: Hystrix wants at least twenty requests in the window; resilience4j wants a hundred. Nineteen failures out of nineteen calls will not trip a Hystrix breaker; there simply isn't enough evidence yet. This is the same instinct as not convicting a node on one missed heartbeat: gather enough witnesses before you act.

Slow counts as failed. The subtlest trap: a dependency can return a perfect 200 OK on every call and still be destroying you, if each one takes ten seconds. It is up and useless. Good breakers watch latency too and trip on a slow-call rate (resilience4j has an explicit slow-call threshold) because a call that ties up a thread for ten seconds does the same damage as one that fails, and refusing to notice is how a "healthy" dependency still takes you down.

Every one of these is the heartbeat bet in new clothes. Too eager — a low threshold, a tiny floor — and a brief hiccup opens the breaker and cuts you off from a service that was about to be fine. Too slow — a high threshold, a long window — and you keep hurling threads at something that died a minute ago. There is no universally right number; there is only the number that fits this dependency's normal behavior, which is why you tune breakers per dependency and watch them like any other detector.

The half-open state has one failure mode nasty enough to deserve its own section, because the naive version turns your recovery into a relapse.

Picture a popular service that just came back. Its breaker's cooldown expires, and in the naive design it half-opens by simply resuming traffic. But while it was open, every one of thousands of clients kept generating requests that failed fast and queued behind the breaker. The instant it half-opens, all of that pent-up traffic floods the barely-recovered service at once and knocks it flat again. The breaker trips back open, the cooldown runs, the flood rebuilds. You've built a slower, meaner retry storm out of the very tool meant to stop one.

The fix is why half-open admits a counted number of calls. resilience4j lets exactly ten probe calls through and holds everything else back until it has a verdict; the older Hystrix let a single request through. A handful of probes is enough to learn whether the dependency can serve, without betting the recovery on the answer. Test gently, decide, then open the floodgates only once you're sure.

A related refinement lives in how breakers punish flapping. A service mesh doing breaker-style ejection — Envoy's outlier detection removes an unhealthy host from the pool — makes each re-ejection last longer than the last. Fail once, sit out a short penalty; keep failing after every reprieve, and the penalties grow. It's a snooze button with an escalating timer, and it stops a genuinely-broken host from cycling through half-open every few seconds.

Below is one breaker wrapping one dependency, and every number is yours. Set the dependency's failure rate and the breaker's threshold, then kill the dependency and watch the sliding window redden, the rate cross the line, and the state snap to open — after which calls fast-fail in microseconds instead of waiting out a timeout, and the spared-calls counter climbs. Wait out the cooldown and watch it go half-open and admit a few probes; heal the dependency first and they close it, leave it dead and it snaps back open.

Then break your own intuition. Try to trip it on just two failures and watch the floor refuse — not enough evidence. Make the dependency merely slow instead of failing, and watch it trip anyway. Set the threshold cruelly low on a dependency that's basically fine, and watch the breaker become the outage by cutting off a healthy service. The machine is honest; the judgment is yours.

It's worth being precise about what the open state actually gives you, because "stops calling" undersells it.

The first gift is speed. A timeout has to wait to fail: the whole budget, every time, holding a thread the entire way. An open breaker fails in microseconds, because it never touches the network; it just returns. Under load that difference is the difference between a thread pool that recycles instantly and one that fills with hostages until you become the outage. Fast-fail is not giving up — it's giving the thread back.

The second gift is a place to put a fallback. Since the open breaker returns immediately, it returns something: a cached value, a sensible default, a degraded answer, an honest "try again later." Recommendations fall back to a bestseller list; a profile page renders without the down-service's badge. The breaker is where graceful degradation gets wired in, and that idea gets a lesson of its own shortly.

And where does the breaker actually run? Two homes. In a client library — resilience4j today, Hystrix before it, the breaker lives inside your process, one instance per dependency, so a payments outage opens your payments breaker without touching search. Or in the service mesh, where a sidecar like Envoy does breaker-style ejection for every service uniformly, no application code required, at the cost of per-dependency nuance. Most large systems run both: coarse protection in the mesh, precise fallbacks in the app. Either way the scope matters — one breaker per dependency, never one for the whole app — because the breaker's job is to keep one sick dependency from taking down everything that shares your process.

• A breaker is a failure detector with a switch. Closed it watches; open it stops calling entirely; half-open it tests. Every hard choice is the heartbeat bet: eager trips cut off the healthy, slow trips keep feeding the dead.
• The threshold is the whole art: a failure rate over a sliding window, a minimum-calls floor so noise can't trip it, and a slow-call rule because up and useless is still a failure.
• Half-open must not stampede. Admit a counted few probes, not the whole pent-up flood, or you relapse the service you were protecting. Punish flapping with growing cooldowns.
• Open buys speed and a fallback. Microsecond fast-fail hands the thread back instead of holding it hostage, and the returned-immediately path is where graceful degradation lives.
• Scope it per dependency, in the library or the mesh. One breaker per dependency, never one for the app; resilience4j inside the process, Envoy ejection at the sidecar, usually both.

Notice the quiet assumption the breaker rests on: that a failing dependency can be cut off cleanly. But a slow dependency does its damage before the breaker ever trips: while those first calls are hanging, they're holding threads from a pool your other, healthy dependencies also draw from, and that pool can be empty before the failure rate ever crosses the line. The breaker protects you from calling a corpse; it does nothing to stop one sick dependency from drowning the resources everything else needs. Containing that blast radius — giving each dependency its own bounded pool so it can only sink itself — is next: Bulkheads.