Timeout Hierarchies

The last section closed on an uncomfortable observation: everything you've built so far quietly assumes that requests eventually return. This section is about the fleet-scale physics of when they don't — and it starts with the only tool that turns maybe never into by Tuesday: the timeout.

You already own the basics. Calling Services Safely: Timeouts, Retries & Backoff taught you that every remote call needs a timeout, that a hung call holds a thread hostage, and that the number should sit just above the dependency's p99. This lesson is the full treatment of the part that one lesson could only wave at: what happens when timeouts nest. A real request crosses four, six, ten hops, and every hop has its own opinion about patience. Those opinions form a hierarchy whether you design one or not — and an undesigned hierarchy has a failure mode where your servers do enormous amounts of work for users who already saw an error page.

Here's the part most teams discover during an incident: your timeout hierarchy already exists. Nobody designed it. It assembled itself out of shipped defaults.

The nginx in front of everything waits 60 seconds for the upstream, and that famous proxy_read_timeout is measured between two consecutive reads, not for the whole response, so a trickling reply can legally take minutes. The Envoy route under it defaults to 15 seconds. And the gRPC client at the bottom of the stack ships with no deadline at all: the documented default is to wait effectively forever, which is why the official guidance says, in bold, to always set one. Sixty seconds, fifteen seconds, infinity: three numbers written by three teams in three config files, none of them aware of the others, none of them derived from anything a user was promised.

That's the accidental hierarchy, and its two standing failure modes are the subject of this lesson: waits that stack into minutes of held threads, and inner calls that outlive the outer request they were serving. Neither is a bug in any one config. The bug is that nobody owns the whole clock.

The repair begins with a change of unit. A timeout is not a per-service knob; it's a slice of one shared resource: the total time this request is allowed to cost, set once, at the top, by a promise to a human — the checkout answers in two seconds; the search feels instant under 400 milliseconds. Everything below is spending that budget.

So the numbers flow in two directions, and keeping them straight is the whole discipline. Budgets flow down: the edge holds 500 milliseconds, spends ~40 on its own work and the network, and can afford at most ~460 for everything beneath; the service spends and passes less again. Each child's allowance must fit inside the parent's remainder, with slack for the hops between — the nesting rule. Percentiles flow up: each dependency's p99, the number you learned to respect in Calling Services Safely: Timeouts, Retries & Backoff, tells you what a hop costs. The two flows meet in the middle, and when they don't fit — when honest p99s sum past the user-level promise, you have learned something a config file can't fix: the design is too slow for the promise, and the answer is caching, parallelism, or a cheaper path, not a bigger number.

Two bookkeeping rules keep the budget honest. Measure it on the monotonic clock — the wall clock's lies were a whole lesson, and a budget that jumps backwards during a sync is worse than none. And count queue time as spent time: a request that waited 300 milliseconds in line has 300 less to live, which means the server should check the remaining budget before starting work: the cheapest possible moment to say no.

One more precision upgrade before the pathology. "The timeout" on a call is usually three different clocks, and conflating them causes real outages.

The connection clock: how long to wait for a socket to open. Short: an unreachable host should fail in tens of milliseconds to a few seconds, and this is the clock that saves you when a whole box is gone.

The request clock: how long from sending to a complete answer. This is the one that should be derived from the budget.

The idle clock: how long between signs of life — bytes read, frames received. This is what that 60-second proxy default actually measures, and it's the right tool for streams and long downloads, where a total-time clock would kill legitimate work. (A fourth pair — per-try versus per-request, once retries enter — waits for the next lesson.)

The classic misconfiguration is using an idle clock where you meant a request clock — the trickling response that never goes quiet for 60 seconds straight and so runs for ten minutes, holding a thread the entire way. Name the clock you're setting, every time.

Now the failure mode the accidental hierarchy manufactures at scale. Picture the chain: edge times out at 500 milliseconds, the service below it at 1000, and the database client (configured by a different team, in a different year) at 2000.

The database has a slow night. At 500 milliseconds the edge gives up, returns an error, and the user starts retyping their search. And nothing else stops. The service keeps waiting on the database; the database keeps grinding the query; at 2000 milliseconds the answer finally arrives, complete and correct, and is thrown away unread. Three-quarters of the work performed for that request happened after the request stopped mattering.

That's timeout inversion, any child allowed to outlive its parent, and its product is zombie work: real CPU, real I/O, real capacity, spent on the already-dead. You've met this shape twice: it's the zombie leader from the consensus lessons and the frozen lock-holder from the lease lesson, reborn as a capacity problem. And here's why it's a fleet killer rather than a nuisance: zombie work is heaviest exactly when the system is slowest, because that's when parents give up. The overloaded database gets extra dead-work precisely at peak: the overload feeds itself. That amplification loop has its own lesson coming; for now, the rule: a child's clock must always be shorter than its parent's remainder. Sum the children under the parent, leave slack for the wire, and audit it whenever either side changes.

Below is the chain with every number under your fingers. Type timeouts at the edge, the service, and the database call; pick a scenario; run it and watch the budget bar drain as the request descends. Stage the inversion first — 500, 1000, 2000 against a slow database — and read the two verdicts: the user did fail fast, and the fleet still paid 1600 milliseconds of zombie work, priced on the counter. Then tighten the children until the waste hits zero, break it the other way (a database allowance so small that healthy queries die), and finally flip propagation on and watch the same chain heal itself: one budget riding the request, every hop inheriting the remainder, the whole tree giving up in the same instant.

Hand-tuning three config files into a valid hierarchy works — until a deploy changes one of them. The industrial fix is to stop shipping timeouts as static opinions and start shipping them as state on the request: deadline propagation.

The mechanics are almost disappointingly small. At the edge, convert the budget into an absolute deadline: not "you have 500 milliseconds" but "this request is dead at 14:03:07.500." Stamp it on the request; the RPC layer carries it in a header and converts it back to a remaining duration at each hop. Every hop now answers one question before doing anything: how much is left? Subtract your own expected spend, pass the remainder down. No hop can ever hold more time than the original caller intended — the inversion becomes unrepresentable.

This is exactly how the mature stacks work, and the reliability literature is blunt about it: rather than inventing a deadline at every tier, set it once, high in the stack, and propagate it. Two corollaries complete the discipline. A request that arrives with its budget already spent gets refused at the door: the queue ate its life, and declining costs nothing while serving it costs everything. And cancellation travels the other way: when the edge gives up, it doesn't just return an error; it cancels the calls beneath it, so the tree stops working the moment the answer stops mattering. Deadlines bound the waiting; cancellation reclaims the work.

One honest caveat: absolute deadlines ride on wall clocks, and you know what wall clocks are. Skew between hops eats budget silently — one more reason the slack in every allowance isn't padding, it's engineering.

• Your timeout hierarchy already exists; nobody designed it. Shipped defaults — 60 seconds between reads at the proxy, 15 at the mesh, forever at the client — assemble themselves into an accident.
• Think in budgets, not knobs. One number, set at the top by a promise to a human; budgets flow down, percentiles flow up, and when they collide you fix the design, not the config.
• Name the clock: connection, request, or idle, three timers on every call, and the idle-for-request mixup is how ten-minute requests happen.
• Inversion makes zombie work, and zombie work peaks exactly at overload. A child's clock must fit inside its parent's remainder, with slack for the wire.
• Propagate the deadline; cancel the tree. Absolute deadline stamped once, carried on the request, checked before work, refused when spent; and when the top gives up, everything below stops.

The timeout answers one question: when do you stop waiting? It says nothing about the next one — what do you do instead? The obvious answer, try again, is also the most dangerous machine in this section: done naively, ten thousand clients asking again is indistinguishable from an attack, and it arrives at the worst possible moment, aimed at your slowest service. That machine, its budgets, and its storms are next: Retries, Budgets & Retry Storms.