Bulkheads

The last lesson gave you a tool for a dependency that has died: the breaker in Circuit Breakers notices the corpse and stops calling it. But it left a gap open, and the gap is the dangerous one. A dependency rarely dies cleanly. Far more often it goes slow, and a slow dependency does its damage in the seconds before the breaker's failure rate ever crosses the line. While those first calls hang, each one is holding a thread or a connection, and it is holding them from a pool that your other, perfectly healthy dependencies draw from too. The pool empties. Now everything is slow, and the breaker that would have saved the sick service never fires, because from the outside your whole application looks sick.

The tool for this is the bulkhead, named by Michael Nygard in the same 2007 book, Release It!, that gave us the breaker. He borrowed it from shipbuilding. A ship's hull is divided into sealed compartments so that a hole in one does not flood the rest and sink the vessel. The idea in software is identical: partition your resources so that one sick dependency can drain only its own compartment, never the shared hull that keeps everything else afloat. The breaker stops you calling a service that is gone. The bulkhead makes sure that a service on its way out can only sink itself.

Before the fix, sit with the failure, because it is sneakier than it looks and almost every service ships with it by default.

Picture an ordinary service. Requests arrive on a pool of a couple hundred worker threads, and to reach the database they all draw connections from one shared connection pool. A connection pool is small on purpose: HikariCP, the standard one for Java, defaults to ten connections, because a database serves far more through a few busy connections than through a thousand idle ones. Ten is plenty, as long as every call gives its connection back in a few milliseconds.

Now one query gets slow. Maybe a reporting endpoint runs a join that used to take fifty milliseconds and now takes five seconds because a table grew. That endpoint is a rounding error in your traffic, so nobody worries. But each of its calls now holds a connection for five seconds instead of fifty milliseconds, and there are only ten connections. It takes a trickle of slow requests to hold all ten. The moment they do, every other request in the building that needs the database — checkout, login, the home page, all of it — lines up behind an empty pool and waits. In one real write-up of exactly this, the query itself took fifty milliseconds and the wait for a free connection took two thousand four hundred. The database was bored. The pool was the wall.

This is the shape of the disaster: the blast radius of a slow dependency is everything that shares a resource with it. The reporting endpoint didn't take down reporting; it took down checkout, because checkout's health was quietly tied to reporting's health through a pool neither of them knew they were sharing. Your monitoring lights up everywhere at once and points nowhere. And notice the cruel timing from the last lesson: the sick dependency's own breaker may be sitting there perfectly closed, because its calls are technically still succeeding, just slowly, while it quietly strangles everything around it.

The fix is the ship's fix: stop sharing the one hull. Give each dependency its own bounded pool, and the blast radius shrinks to that pool.

Concretely: instead of one connection pool of ten that everything fights over, carve it up. Payments gets its own small pool. Search gets its own. The reporting endpoint gets a tiny one of its own. Now when reporting goes slow and holds every connection it has, it exhausts its compartment and its calls start failing, and that is the whole of the damage. Checkout draws from a different pool that reporting cannot touch, so checkout keeps working while reporting drowns. You have traded one big failure for one small, contained one, on purpose.

The word that makes it a bulkhead and not just "more pools" is bounded. A pool that can grow without limit is not a compartment; it is an open deck, and under load it will happily allocate ten thousand threads and take the machine down a different way. The bound is the wall. It says: this dependency may consume this much of the ship and not one plank more, no matter how badly it misbehaves. When the compartment is full, the next call is turned away immediately rather than allowed to widen the breach.

And the compartments come at every scale, which is worth seeing now because it repeats all the way up. Nygard's own examples climb a ladder: separate thread pools inside a single process, separate connection pools to a database, separate processes on a machine so a crash takes one down and not the rest, separate machines or clusters so a hardware fault is contained, and at the very top a whole isolated copy of the system serving a slice of users. It is the same move — draw a wall, bound what is inside it — applied at a bigger and bigger grain.

When you reach for a real bulkhead in a real library, you meet a fork that looks like an implementation detail and is actually a deep one. There are two ways to bound concurrency, and they differ in the one thing you care about most under failure: whether you can let go of a call that has hung.

A semaphore is the cheap wall. It is just a counter of permits. Before a call runs it takes a permit; when the call returns it gives the permit back; when the permits are all out, the next call is rejected on the spot. That is the entire mechanism, and it costs almost nothing — a few nanoseconds on the thread that was already running the call. resilience4j's semaphore bulkhead defaults to twenty-five permits and a wait of zero, meaning when the wall is full it fails you now rather than parking you in a line. The catch is exactly that the call runs on your own thread. A semaphore can limit how many slow calls pile up, but it can never interrupt one. If the dependency hangs, the threads already inside stay hostage until the underlying call times out on its own. It bounds the bleeding; it cannot stop it.

A thread pool is the stronger wall, at a price. Each dependency gets its own small pool of worker threads, and a call is handed off to that pool instead of run in place. Because the call now runs on a thread you own and can throw away, you can put a hard timeout on it: when it blows the deadline, you abandon the worker, and — this is the point — the caller's thread was handed straight back the moment it delegated, so the caller is never held hostage at all. That is why Netflix's Hystrix made thread-pool isolation its default: a bounded pool of ten threads per dependency, and, tellingly, no waiting room — its default queue holds nothing, so the eleventh concurrent call is rejected immediately instead of buffered. The cost is real, though: every call pays a thread hand-off and a context switch, and one pool per dependency across dozens of dependencies is a lot of threads. So the rule of thumb the libraries settled on: thread pools for network calls you need to be able to time out, semaphores for cheap in-memory calls that are too high-volume to afford the hand-off.

A wall in the wrong place is as bad as no wall, so the last question is the practical one: how many slots does each compartment get? Too large and it isn't really a bound — one dependency can still eat most of the ship. Too small and you reject healthy traffic that the dependency could easily have served. You have met this shape before; it is the same knife-edge the breaker's threshold sat on in the last lesson, and the timeout's budget before that.

There is a clean way to reason about it, and it comes from queueing theory. Little's Law says the average number of requests in flight equals the arrival rate times the time each one takes. So the concurrency a healthy dependency actually needs is just its throughput times its latency. If payments serves two hundred calls a second and each takes fifty milliseconds, then on average ten calls are in flight at once, and a pool of ten is exactly enough for the steady state. Size the compartment to that number plus a little headroom for bursts: big enough that normal traffic never waits, small enough that a hang cannot consume more than this dependency's fair share of the ship.

And here is the shift in thinking the bound forces on you. In the shared-pool world, running out of a resource meant waiting — parking a thread against an empty pool for two thousand milliseconds, hoping. A bounded bulkhead replaces that wait with an immediate, honest rejection. That feels worse and is almost always better: a call rejected in a microsecond returns a thread you can reuse, shows a fallback, and keeps the rest of the ship dry, while a call parked against a dead pool just adds itself to the pile. Failing fast when the compartment is full is not the bulkhead giving up. It is the bulkhead doing its job.

Reading about a blast radius is one thing; watching healthy traffic die because of a dependency it never called is another. Below are three dependencies — payments, search, and profile — taking steady traffic, and the pools they draw from are yours to arrange.

Start in shared mode, where all three draw on one common pool, and make payments hang. Watch its slow calls hold every slot, watch search and profile find nothing free and start getting rejected though nothing is wrong with them, and watch the blast-radius meter climb to all three. Then flip to bulkheads, give each dependency its own compartment, and hang payments again: this time payments fills and rejects only its own pool while search and profile sail through, and the blast radius stays at one. Set the pool sizes and the request rate by hand, and toggle thread pool against semaphore to watch the one real difference — the thread pool abandons a hung call at the timeout and frees the slot, while the semaphore keeps the slot pinned until the call finally comes back. The containment is the whole lesson, and here it is something you can cause and undo with your own hands.

The ship metaphor comes with its own warning built in, and it is the most useful part of the whole picture. The Titanic had watertight compartments. What it didn't have was compartments that reached high enough. The bulkhead walls stopped a few decks up, so when the flooded bow tipped down, water simply rose above the top of one wall and spilled over into the next compartment, and then the next, in sequence. The isolation was real; it just wasn't sealed all the way up, and unsealed isolation floods in slow motion instead of all at once.

Software bulkheads fail the same way, and the mistake is easy to make on paper. You give payments and search their own separate thread pools and congratulate yourself on the isolation, but both pools hand their work to the same shared database connection pool underneath. Or both run on the same single-threaded event loop. Or both live in the same process, competing for the same CPU and the same garbage collector. You drew two compartments, but they share a floor, and when the sick dependency drains that shared floor the water spills over exactly like it did on the ship. A bulkhead is only as sealed as the deepest resource its compartments share. Isolating threads while sharing the connection pool underneath is isolation theater. Trace the call all the way down and put the wall at the deepest thing two dependencies contend for.

Follow that instinct up the ladder and you arrive, eventually, at giving whole groups of users their own private copy of the system — separate pools, separate databases, separate everything, so a failure is walled inside one slice and cannot reach the rest. Amazon draws exactly this line: bulkheads for data they call partitions, and bulkheads for whole services they call cells, deliberately spreading users across them so any one failure touches only a fraction. That is the same watertight-compartment move at the scale of an entire system, and it earns its own lesson later in Cell-Based Architecture & Deployment Stamps. For now, hold the through-line: the wall is the same idea whether it bounds ten threads or ten thousand users.

• A slow dependency is more dangerous than a dead one. It does its damage in the seconds before a breaker can trip, by holding shared resources that healthy dependencies need. The breaker cuts off the corpse; the bulkhead contains the one still dying.
• The blast radius is everything that shares a resource. One slow reporting query took down checkout because they shared a ten-connection pool. Isolation shrinks the radius to a single compartment.
• Give each dependency its own bounded pool. Bounded is the word that matters: the limit is the wall, and a full compartment rejects the next call immediately instead of widening the breach.
• Thread pools can time a hung call out; semaphores cannot. Hand the call to its own pool (Hystrix's default ten, no queue) when you need to abandon it and reclaim the thread; use a cheap semaphore (resilience4j's default twenty-five) for high-volume in-memory calls. Size each pool by Little's Law: throughput times latency, plus headroom.
• A bulkhead is only as sealed as the deepest resource it shares. Isolated thread pools over one shared connection pool spill over the top like the Titanic's short walls. The move repeats up the ladder, all the way to isolated cells.

One thing the bulkhead does not answer, though, is what happens at the wall. When a compartment fills and starts rejecting calls, the rejection is the right move locally, but it leaves a question sitting upstream: the caller just got told no, I'm full — so what should the caller do? Hammer harder and you have a retry storm against a wall. Wait politely and you need a way to be told how long and how much. Turning a full compartment's blunt rejection into a signal that flows back up the chain and asks the sender to slow down, rather than a request simply dropped on the floor, is the next tool: Backpressure.