Checkpoint: Resilience & Overload Control

The codelab ended with a bill: ten decisions drawn from the whole of resilience and overload control. This is where it's paid.

Ten lessons ago, "the system stays up" was a hope you had to take on faith. Since then: every wait got a deadline, every retry got a budget, every dying dependency got a breaker, every pool got a wall, every full queue learned to push back, every overload learned to shed, the tail got hedged, the cascade got named, and then you broke a real system on purpose and watched a defense hold. The exam below never asks for a definition. It hands you a room with something on the line and asks what you do.

One compression before the exam. Every lesson here is the same law in a different costume:

Everything you depend on will one day be slow, then dead, and you will not be able to tell which. So decide your response in advance: bound every wait, cap every retry, wall off every pool, and shed the load you can't serve — so one dependency's bad day stays one dependency's bad day. And never trust a defense you haven't triggered on purpose.

• Timeout Hierarchies: the deadline. Every wait is bounded, and the budget shrinks as it flows inward, so no one works on a request its caller has abandoned.
• Retries, Budgets & Retry Storms: the amplifier. Retries multiply down a stack into a storm; a budget, jitter, and one retry layer keep help from becoming the outage.
• Circuit Breakers: the detector. Stop calling a corpse — one breaker per dependency, tripping on errors and on slowness, fast-failing to a fallback.
• Bulkheads: the wall. A bounded pool per dependency, so one sick dependency can only sink itself, and 'full' is a feature.
• Backpressure: the signal. A bounded queue that, when full, tells the sender to slow down instead of hiding a backlog as latency until it dies.
• Load Shedding & Graceful Degradation: the valve. When you can't push back, drop the least-important work at the door and serve a worse-but-alive answer for the rest.
• Tail Latency: Living at p99: the honest metric. The average lies; fan-out amplifies the slowest, and a hedged request races the tail.
• Cascading Failures: the whole failure. A local slowness climbs the call graph into a metastable outage that removing the trigger won't cure; no single tool stops it.
• Chaos Engineering: the proof. A defense you've never triggered is a hope; inject the failure, in the small, and find out.

Read the law once more and notice its two halves. The first half is the machinery: a deadline, a budget, a wall, a valve. The second half is the judgment: containment is a decision you make before the incident, and the only way to know it works is to cause the failure yourself.

Half the exam hands you a misbehaving dependency and asks what you reach for, because that's the form this knowledge takes in a real incident. The whole method is three questions and a standing rule:

Question one: is the dependency slow, or dead? Slow means bound the wait — a timeout sized to your budget, and a hedged request to race the tail when the slowness is only the unlucky few percent. Dead or failing means stop calling it — a circuit breaker that fast-fails to a fallback, so you're not throwing threads at a corpse.

Question two: is the failure spreading from one dependency to the rest of you? Then isolate it. A bulkhead gives each dependency its own bounded pool, so a slow one can exhaust only its own compartment and the rest of the ship stays dry.

Question three: are you the one being overwhelmed by inbound load? If you can push back, send backpressure up the chain and let the sender slow to your real rate. If you can't — the sender is the public internet — then shed: drop the least-important requests at the door, protect the critical ones and the health check, and degrade the rest to a cheaper answer.

The standing rule: every defense has a knob, and both ends are wrong. Too loose and it never triggers; too tight and it rejects healthy work. A timeout longer than your caller's, a breaker that never trips, a pool sized for the peak — each is a defense in name only. And no single tool stops a cascade: layer them, because a cascade finds the first link you left unprotected. Then prove the whole stack the only way that counts — inject the failure and watch it hold.

Ten scenarios from the rooms where this section earns its keep: a timeout review, a post-incident on a retry storm, a breaker that guarded too much, a starved connection pool, a queue that ate its own memory, a goodput cliff, a fan-out that betrayed its leaves, a cascade that wouldn't die, a reckless chaos plan, and one 'non-critical' dependency that wasn't. Every wrong option is a real wrong turn these ten lessons taught you to refuse. Pass is seven of ten; miss one and the verdict names the exact lesson to revisit.

If this section landed, here is the capability it certifies.

In a design review, you can hear "we retry on failure" and ask the three questions that decide whether it's a safety net or a loaded gun: at how many layers, under what budget, with jitter? You can look at a shared connection pool and see the blast radius before it fires. You can refuse a timeout that's longer than its caller's, a breaker that guards the whole app, and an unbounded queue sold as 'we never drop data,' with the same confidence.

In an incident, you can read a dependency that's slow-but-up and know why the error-rate breaker never saw it. You can watch goodput collapse under overload and name the zombie work and the retry loop keeping it on the floor. You can see a cascade that won't recover at a load it used to handle and know it's metastable, so you throttle far below the tipping point and ramp back with cold caches in mind.

And under all of it, you know the two-part law: bound, cap, wall, and shed so a failure stays contained — and then break it on purpose, because a defense you've never triggered is a hope, not a fact. That reflex is assumed from here on.

• The law: everything you depend on will be slow, then dead, and you won't know which; bound the wait, cap the retry, wall the pool, shed the rest, and prove it by breaking it. Ten lessons, one sentence.
• The nine defenses, one line each: a deadline on every wait · a budget on every retry · a breaker on every dying dependency · a wall around every pool · a signal when the queue is full · a valve to shed what you can't serve · a hedge for the tail · the cascade that no single tool stops · and the experiment that proves the rest.
• The reflex: slow or dead, spreading or not, overwhelmed or not. Timeout and hedge, or breaker and fallback; bulkhead to isolate; backpressure or shed to survive inbound load. Every knob is wrong at both ends.
• Passing means permission. Seven of ten says you can keep a system alive under a dependency's worst day, and everything ahead assumes exactly that.

Now notice what every lesson in this section quietly assumed: that you could see the failure — the error rate, the p99, the goodput, the steady-state line. You cannot defend what you cannot measure, and you cannot promise what you cannot define. The next section turns from keeping the system up to knowing whether it's up at all, and to the numbers you put your name on: what to measure, what to promise, and how much failure you're allowed to spend. That discipline begins with SLIs, SLOs & Error Budgets.