Load Shedding & Graceful Degradation

Backpressure ended on an honest admission. You can push back on your own services all the way to the edge, but the edge is a stranger's phone that honors no signal, and when the source simply cannot be slowed, pushing back is over. You are left with more work than you can ever finish, and something has to give. The only question left is whether you choose what gives, or the overload chooses for you, because it will, and its choices are terrible.

This lesson is the last tool in the overload kit, and it is really two tools that share one idea. Load shedding drops some requests entirely so the rest can be served. Graceful degradation keeps every request but hands it a cheaper answer: stale instead of fresh, a list instead of a personalized list, a static page instead of a live one. Both come from the same refusal to pretend: when you cannot serve everyone well, a fast no and a cheaper yes both beat a slow death for everybody. Reduced service beats no service. The whole art is choosing, deliberately and in advance, exactly what to give up first.

To see why shedding isn't optional, you have to watch what happens to a server that doesn't do it, because the failure is worse and stranger than "it gets slow."

Draw the useful work a server actually completes — its goodput — against the load offered to it. At first the two rise together: more requests, more work done. Then you hit the server's capacity, and here is where intuition betrays you. You would expect goodput to plateau at capacity and stay there. It doesn't. Past the peak it falls off a cliff, collapsing back toward zero, so that a server hammered hard enough completes almost no useful work at all while running flat out at 100 percent busy.

The reason is that every request costs something the moment it arrives, whether or not it is ever answered. It takes a thread, a slot in a queue, memory, CPU just to parse and track. Under overload the queue fills with requests that will sit there so long the client gives up before they're reached, and the server dutifully spends its scarce resources computing answers nobody is waiting for anymore — the zombie work from Timeout Hierarchies, now at the scale of the whole machine. Then the clients that gave up retry, adding fresh load exactly when there's none to spare, and you have the self-sustaining collapse from Retries, Budgets & Retry Storms. The server is busiest precisely when it is least productive.

This is the sentence to keep: if you don't shed load, the system sheds it for you — through timeouts, dropped connections, retries, and total collapse, which is the worst possible way to do it. Shedding is nothing more than choosing to turn that cliff back into the plateau you expected in the first place. The goal, in Google's words, is that a server built to serve some rate keeps serving that rate no matter how much excess is thrown at it, and never falls over.

The mechanics of shedding are simpler than they sound, and one detail matters more than all the rest: shed at the door, before you spend anything.

Load shedding is the act of cheaply rejecting work instead of spending limited resources on it. The moment a request arrives, before it takes a worker thread or a database connection, you decide to accept or reject it — admission control at the entrance, not triage in the back. A rejected request should cost almost nothing: an immediate 503, not a slow error after a timeout. This sounds obvious until you notice the trap hiding in it. If saying no is expensive — a full connection setup, a fat error page, a slow socket teardown — then shedding under load just adds a second expensive thing to do, and it accelerates the collapse instead of stopping it. Amazon's teams push the rejection as low as it will go: the operating system's firewall can drop an excess connection far more cheaply than any application process can, and the load balancer can turn a request away before your service ever sees it.

The second detail is when to shed, and the honest answer is: sooner than feels comfortable, and on the right signal. Waiting until CPU hits 100 percent is waiting until you're already over the cliff, because CPU is a lagging signal. Better to watch a leading one — the depth of the queue, or how long requests are waiting in it. Facebook's servers borrow a trick from the networking world you met as bufferbloat in Backpressure: an algorithm called CoDel watches the time a request spends waiting in the queue, and when that wait crosses a small target it starts shedding, because a growing wait is the earliest honest sign that arrivals are outrunning service.

The third: don't guess your capacity, measure it. A fixed "1,000 requests per second" cap, the kind from Rate Limiting: Token Bucket, Leaky Bucket, Windows, is a guess made in advance, and the real limit moves with the workload — a heavier mix of requests lowers it, a warm cache raises it. Netflix's servers borrow TCP's congestion-control idea and adjust their concurrency limit continuously from observed latency: when latency starts to climb, the safe in-flight count has been passed, so tighten. The limit finds itself, and the shed happens in well under a millisecond, fast enough not to feed a retry storm.

Shedding something keeps you alive. Shedding the right something is the entire art, because a dropped request is a real cost to a real person, and some of those people matter more than others to the survival of the whole.

The move is to rank requests by importance and drop from the bottom up. Google's infrastructure gives every request one of four criticality levels — from the top: CRITICAL_PLUS for the requests whose failure hurts most, CRITICAL for the normal user-facing default, SHEDDABLE_PLUS for work that can tolerate some failure, and SHEDDABLE for batch and background jobs that can be retried an hour later. Overload sheds them in that order, and the sharp part is how: each level is dropped at a different utilization. Sheddable traffic might start being turned away at 50 percent load, while critical-plus is only refused at 100 percent, right at the edge. As pressure rises you peel the layers off one at a time, cheapest first, so the checkout survives an overload that costs you the recommendation carousel.

Two refinements make this real. First, criticality propagates, exactly like the deadline in Timeout Hierarchies: when a critical request calls three backends, those calls inherit its criticality, so the whole tree of work it triggers is protected together rather than being throttled halfway down. Second, the single most important request to protect is the one you'd never think of — the load balancer's health check. Shed those pings to save resources and the balancer concludes your server is dead, pulls it out of rotation, and pours its share of traffic onto your already-drowning neighbors. You will have shed yourself out of existence. The health check outranks everything.

And one last, counterintuitive move for the moment you're truly underwater: be unfair on purpose. A queue is normally first-in-first-out, which feels just. But under heavy overload the requests at the front are the oldest, and the oldest have been waiting so long their clients have already given up — serving them is pure zombie work. Facebook's servers flip the queue to last-in-first-out when overloaded, serving the newest requests first, because those are the only ones still young enough to have someone waiting for the answer. Fairness gets everyone a timeout; a deliberate unfairness gets the freshest requests an actual response.

Dropping a request gives its owner nothing. Often you can do better than nothing, and that is the other tool: keep serving everyone, but serve them something cheaper. This is graceful degradation, and it is the systematic, planned version of the single fallback you met behind an open breaker in Circuit Breakers — not one backup answer for one dead dependency, but a whole ladder of worse-but-alive answers the system can climb down as it gets more stressed.

The rungs are worth having in your head, because they're the menu you'll actually pick from in an incident. Serve stale cache with an honest age limit instead of a fresh read, so a slightly-old answer stands in for a slow database. Reduce fidelity: lower-resolution images, ten search results instead of fifty, the ranking without the expensive personalization pass. Turn an expensive feature off with a flag — drop the "customers also bought" row and the page still sells the product. Fall back to read-only, refusing writes but parking them in a durable queue to replay when you recover, which is the pocket-sized loop from Offline-First Sync used as a pressure valve. And at the far end, serve a static fallback page from the edge when the origin is down entirely, so users see a real page instead of an error.

The word for this is brownout, borrowed from power grids: when demand outstrips supply you dim the lights across the city rather than cut the power to half of it. A dimmed service keeps everyone partially served, which is usually a better outcome than a bright service for some and darkness for the rest. But notice what every rung really is: a product decision, made in advance and in calm. Someone has to have decided, before the incident, that a homepage without the personalized row is acceptable and a checkout without a fraud check is not. Graceful degradation is only graceful if the tradeoffs were chosen deliberately, long before the night you need them.

Everything above is one picture in motion: a service under a rising flood, and a dial that decides who it lets down and how. Below, the load climbs and the policy is yours.

Start with shedding off and push the offered load past capacity. Watch goodput rise to the capacity line and then fall off the cliff toward zero as retries and timed-out zombie work eat every thread — the server at its busiest and least useful. Now turn shedding on and watch the same flood produce a flat line pinned at capacity: the excess is refused cheaply at the door and the machine spends everything it has on work that will actually be delivered. Split the traffic into priority classes and watch the critical lane stay fully served while the sheddable lanes are dropped first, peeled off in order as load climbs. Flip a lane from drop to degrade and watch those requests come back to life in a cheaper lane instead of failing. Then try, with shedding on, to make goodput collapse. You won't be able to. That flat green line is the entire point of the lesson.

Shedding done carelessly is worse than no shedding at all, because it burns effort on the way down. Four traps catch almost everyone at least once.

Shedding the health check. Covered above, and worth repeating because it is the most common self-inflicted outage in the whole pattern: protect the load balancer's pings above every other request, or your shedding removes you.

Expensive rejection. If your no costs nearly as much as your yes, shedding doesn't save you, it just changes what you die of. Load-test far past the point where goodput plateaus, because the cost of a rejection only shows up out there, and sometimes it turns out to be cheaper to finish a request than to reject it badly. Push the no down to the load balancer or the firewall where it's nearly free.

Letting clients retry the shed. A 503 that a client immediately retries has not reduced your load; it has reshaped it into a tight loop that rebuilds exactly what you just dropped — the retry storm from Retries, Budgets & Retry Storms wearing a shed response as a disguise. Shed responses need to discourage the retry: a clear back-off signal, and clients that throttle themselves when they see refusals piling up.

Never testing it. The shed path is code that runs only during an incident, which means the first real execution happens on your worst day, under maximum stress, with everyone watching. Untested shedding fails in the exact moment it exists for. The only fix is to run the overload on purpose, in advance, and watch the plateau hold — which is the whole spirit of the chaos work later in this section.

• When you can't serve everyone, choose what gives. Load shedding drops some requests entirely; graceful degradation keeps all of them but serves a cheaper answer. Both beat a slow death for everybody. Reduced service beats no service.
• Unshed overload collapses goodput to zero. Every request costs resources even when its answer is thrown away, and retries pile on, so a busy server does almost no useful work. Shedding turns the cliff back into a plateau at capacity.
• Shed at the door, cheaply, early, and measured. Reject before you spend a thread, with an immediate 503 pushed as low as the load balancer or firewall; trigger on a leading signal like queue wait, not lagging CPU; and find the real limit from latency (Netflix) rather than a fixed guessed rate.
• Priority is the whole art. Rank requests (Google's four criticality levels), drop the least important first as utilization climbs, let criticality propagate to downstream calls, protect the health check above all, and under true overload serve newest-first so the freshest requests still make their deadline.
• Degrade when you can, and decide the tradeoffs in advance. Stale cache, reduced fidelity, features off, read-only with a write queue, a static page: a brownout that dims the service for everyone. What "good enough" means is a product decision made before the incident, not during it.

Look back at what shedding actually drops. It drops precisely the requests that would have sat longest in the queue and blown their deadlines — the requests that live in the slow tail of your latency distribution. Shedding, underneath, is a way of defending that tail: cut off the worst-case waits before they happen. But the tail is worth understanding on its own, because even a perfectly healthy system has one, and at scale the slowest one percent quietly decides how fast your whole system feels. That is the next lesson: Tail Latency: Living at p99.