Scaling & Reliability Patterns

You designed the right API shape (L244), added rate limiting (L245), wired CI/CD for evals (L246), and versioned every artifact (L247). The feature works. Then it gets popular — traffic doubles on launch day — or your model provider has one of its routine bad afternoons, and suddenly your beautifully-built app is timing out or down.

This lesson is about the two things that keep a production AI service standing: scaling (handling more load) and reliability (staying up when the pieces you depend on fail). They sound like one topic but they are two — a system can scale beautifully and still fall over the instant a provider returns a 503, and a system can be bulletproof at 10 req/s and melt at 100. You need both.

Here is the reframe that makes all of it tractable: your code is the easy part. A stateless web tier scales by adding boxes. The hard, expensive, failure-prone part is the model — a GPU you can't trivially clone, or a third-party API that rate-limits and goes down on someone else's schedule. So scaling and reliability for AI are mostly about managing one slow, finite, unreliable dependency.

In this lesson:

• Why capacity is concurrency ÷ latency (Little's Law) — and why the GPU is the bottleneck
• Autoscaling LLM workloads on the right signal (queue depth, not CPU) — and the cold-start trap
• The reliability toolkit: timeouts, retries with backoff + jitter, circuit breakers, bulkheads, load-shedding
• Redundancy: multi-provider and multi-region failover (and why extra API keys don't add capacity)
• Durable execution — keeping long agent runs reliable without re-paying for every token
• Capacity planning and SLOs / error budgets

Scope: this lesson is the resilience of the system under load and failure. It builds on fallbacks, retries & circuit breakers for a single call (L234), async/queue API shapes (L244), rate limits & quotas (L245), and the model gateway (L219). It defers how you see all this in production — metrics, logs, traces, dashboards, alerting — to the section finale, L249: Wiring Up Observability End-to-End. You can't fix what you can't measure, but first you need the patterns worth measuring.

Production failures of an AI service come in exactly two flavors, and the fixes are different.

1. Can't keep up (a scaling problem). Demand exceeds capacity. Requests queue, latency climbs, p95 blows past your SLO, and eventually requests time out or get dropped. Nothing broke — you simply don't have enough throughput. The fix is capacity: more replicas, more provider quota, batching, caching, or shedding the excess gracefully.

2. Falls over (a reliability problem). A dependency fails — the provider returns 503, a region goes dark, a tool call hangs, a deploy ships a bad model. Load might be totally normal; one link in the chain broke and took the request (or the whole service) with it. The fix is resilience: timeouts, retries, circuit breakers, redundancy, graceful degradation.

A useful mental test: if you doubled your servers, would the problem go away? If yes, it's a scaling problem. If no — if one bad dependency still takes you down — it's a reliability problem.

Most real incidents are a nasty interaction of the two: a provider slows down (reliability), every request now holds a connection longer, your concurrency fills up, and a service that was fine at this load suddenly can't keep up (scaling). That coupling is exactly why this lesson treats them together — and why the interactive at the end lets you trigger both and watch them feed each other.

Before any pattern, internalize the one equation that governs throughput. It's Little's Law, and for a request/response service it reduces to:

throughput (req/s) = concurrency ÷ average latency (s)

If each in-flight request takes 1.5 s and your system can hold 96 concurrent requests, you can serve 96 / 1.5 ≈ 64 req/s — no matter how many idle CPUs you have. To go faster you have only three levers: raise concurrency (more replicas, more GPU), lower latency (smaller model, shorter outputs, caching, speculative decoding — L210), or shed load so you never exceed what you can serve. Run it backwards for capacity planning: to handle 200 req/s at 1.5 s you need 200 × 1.5 = 300 concurrent slots.

The stateless tier is the easy half. Your API servers should hold no per-user state — sessions live in Redis or the DB (L229), so any replica can serve any request. That makes the web tier horizontally scalable: put a load balancer in front and add identical, disposable replicas. This part is a solved problem.

The model is the hard half — and it's the bottleneck. An LLM request is heavy and GPU-bound. You can't just ‘add a thread’; you add GPUs, which are scarce and expensive, or you lean on a provider whose capacity you don't control. The decisive, counter-intuitive fact for everything that follows:

A serving GPU sits at ~100% utilization whether it is efficiently serving 50 concurrent requests or saturated and dropping them. GPU/CPU utilization tells you almost nothing about whether you have headroom.

That single fact breaks the default autoscaler — which is the next section. (Two big throughput wins on the model itself, continuous batching and KV-cache reuse, you already met in L208/L209/L211; they raise the concurrency term of Little's Law for free.)

The reflex is to autoscale on CPU or GPU utilization, the way you'd scale a normal web service. For LLM inference this is meaningless — as we just saw, the GPU reads ~100% whether it's healthy or drowning, so a CPU/GPU-utilization HPA either never scales or scales on noise.

Scale on a signal that actually tracks backlog: queue depth (requests waiting), inter-token latency / TTFT, or concurrent requests in flight. These rise the moment you start falling behind, before users feel it. In Kubernetes the common stack is KEDA (or a custom-metrics HPA) reading those numbers from vLLM/Prometheus:

Two scheduling rules that matter as much as the metric:

• Scale up fast, scale down slow. Spin new replicas the moment the queue grows (1–3 min), but drain down conservatively — a stabilization window, one pod at a time, behind a PodDisruptionBudget — so a brief dip doesn't kill capacity you'll need 30 seconds later.
• Cold starts will ruin you if you ignore them. A new LLM replica must load tens of GB of weights. Pull them from the internet and your ‘2-minute’ scale-out is a 2-hour one; bake the model into the image or read from fast local/network storage and it's seconds-to-minutes. Until the new replica is warm, you're still over capacity — which is why you scale up early and keep a little headroom.

Done right, autoscaling tracks demand instead of provisioning for peak, which is reported to cut GPU cost by 80%+ for bursty traffic — and newer KEDA can even scale to zero for idle models. If you're on a managed API (OpenAI/Anthropic/Bedrock), you don't run pods — but the same idea applies as concurrency limits + a request queue in your app tier, so you shape load to the provider's quota (L245) instead of hammering it into 429s.

Now the falls-over half. The first three tools are non-negotiable on every call you make to a model or a tool.

1. Always set a timeout. A request with no timeout can hang forever, holding a connection, a worker slot, and memory — and a few hung calls to a slow provider will fill your concurrency and take the whole service down (that's the scaling/reliability coupling again). Set an explicit, per-call timeout (and a separate, longer one for streaming).

2. Retry — but only transient errors. A 429 (rate limit), 503, 500, or a network blip is worth retrying; a 400 (bad request) or a content-moderation refusal is not — retrying it just wastes time and money.

3. Back off exponentially, and add jitter. If every client retries after exactly 1 s, 2 s, 4 s, all your clients retry in lockstep and hammer the recovering service in synchronized waves — a retry storm. Jitter (randomizing the delay) spreads them out; AWS's Builders' Library shows it's the difference between a thundering herd and a smooth recovery. Reported impact: exponential backoff with jitter cuts retry-storm load by ~87% versus fixed intervals, and lifts success on transient failures from roughly 68% to 94%.

The retry trap you must respect: retries fix transient failures; they are poison during overload. If the service is slow because it's saturated, every retry adds more load — you amplify the very problem you're trying to survive. (In the interactive, turn on Retries while overloaded and watch goodput fall.) Two guards: a retry budget (cap retries as a fraction of total traffic), and the circuit breaker in the next section.

One more rule: retries demand idempotency. Retrying a read is safe; retrying a call that charged a card or sent an email can double it. Pass an idempotency key so the downstream dedupes a retried-but-already-applied request (you saw this for your own API in L244).

Retries protect a single call. These three protect the whole system from one bad dependency or a flood.

Circuit breaker — stop hammering a service that's down. It wraps a dependency in three states: Closed (normal — calls flow), Open (too many recent failures → fail fast without even calling, for a cooldown), and Half-Open (after the cooldown, let one trial call through; success → close, failure → re-open). The point is twofold: you fail in milliseconds instead of waiting for timeouts (so your workers don't pile up), and you give the struggling dependency room to recover instead of retry-storming it. This is the system-level partner to L234's per-call resilience.

Bulkhead — isolate the blast radius. Named after a ship's watertight compartments: cap the concurrency allowed to each dependency. If your summarizer tool gets slow, a bulkhead of, say, 20 concurrent calls means it can consume at most those 20 workers — the other 180 keep serving chat. Without it, one slow integration drinks your entire connection pool and sinks everything.

Load-shedding vs. backpressure — choose how to fail under overload. When demand simply exceeds capacity, you have two graceful options and one bad one:

• Load-shed: proactively drop the excess immediately with HTTP 429 so the requests you do accept stay fast. You serve fewer users, but well — graceful degradation.
• Backpressure: push the ‘slow down’ signal back to the caller (429 + a Retry-After header, or a bounded queue) so producers ease off instead of piling on.
• The bad option — an unbounded queue: it feels safe but under sustained overload it just inflates p95 until everything times out anyway. A queue is for smoothing bursts, not for absorbing a load you fundamentally can't serve. For sustained overload, shedding beats queueing — and the interactive lets you prove it to yourself.

The throughline of all three: never collapse. A service that serves 70% of traffic fast and rejects 30% cleanly is infinitely better than one that accepts 100% and dies.

Everything so far assumed your system is the weak link. But if you call a hosted model, your reliability is capped by your provider's — and provider outages are no longer rare events. One status tracker logged 47 incidents across major AI providers in December 2025 alone, including ~20 for Anthropic and ~22 for OpenAI that month. If your app has a single hard dependency on one model endpoint, you have inherited their downtime as your own.

The fix is redundancy — but you have to do it right. First, a myth to kill: adding more API keys does not add capacity. Providers enforce rate limits at the organization / project / model-family level, so extra keys under the same org share one pool. Real headroom comes from independent quota pools: a different provider, a different region, a separately-provisioned project, or provisioned throughput (reserved capacity you pay for) — each carries its own limit, so spreading load across them genuinely adds capacity and removes a single point of failure.

Reliability here comes in two layers:

• Provider-layer failover: primary provider returns 503 / times out → reroute the same model class to another provider or region (e.g. Anthropic API → Bedrock → Vertex for a Claude model).
• Model-layer fallback: the model itself is the problem (a 429, a context-length error, a refusal) → fall back to a different model (a smaller/cheaper one, or a competitor).

You met the place this lives in L219: the model gateway. A gateway turns fallback chains, weighted and latency-based load-balancing, health checks, and retries into configuration instead of per-call code — at the cost of being a control point you must itself run highly-available (multiple instances / zones). The trade-off to design for: failover must be transparent to the user, but a fallback model may be weaker, so degrade visibly enough that you don't silently ship worse answers.

Everything above is tuned for a single, short request. But a growing slice of AI work is long and multi-step: a deep-research agent, a multi-tool workflow, a batch job that runs for minutes or hours (the async API shapes from L244). For these, a new failure mode dominates: if the process crashes at step 9 of 12, you lose everything — and re-running means re-paying for every token you already spent.

Most agent frameworks treat execution as ephemeral — in-memory, gone on a crash. Durable execution fixes this by making the workflow's state persistent and replayable:

• Checkpoint after every step — after each LLM call returns and after each tool returns — so a crash loses at most one step of progress.
• Replay to resume. The engine records each step's result to a durable history; after a crash it replays that history to rebuild in-memory state and continues at the exact step it failed on, reusing recorded results instead of re-calling the model (so you don't re-pay for tokens).
• Idempotency is the prerequisite. Replay re-enters your code, so any step with a side effect (a tool that writes to a DB, sends mail, charges money) must carry an idempotency key derived from (workflow_id, step_id), and workflow code must be deterministic (no random()/now() in the workflow body — push non-determinism into recorded steps).

This is at-least-once execution with idempotent activities, and it's why durable-execution engines — Temporal, AWS Step Functions, Restate, DBOS, Inngest, and framework checkpointers like LangGraph's — have become the 2026 backbone for production agents. It's the reliability layer that makes a 20-minute agent run survivable.

Reliability isn't a vibe; it's a number you commit to and budget against.

Plan capacity with Little's Law and the provider's limits. Estimate peak req/s, multiply by average latency to get the concurrency you must support (200 req/s × 1.5 s = 300 slots), then check that against your provider's TPM/RPM quotas (L245) and your replica budget. The binding constraint for most teams isn't CPU — it's the provider rate limit, which is why provisioned throughput and multi-pool redundancy exist.

Set SLOs, and tier them. An SLO is a target like ‘p95 latency < 4 s and ≥ 99.5% of requests succeed, measured monthly.’ Track p95/p99, not the average — averages hide the tail where your worst experiences live. Tier them: strict SLOs for the core journey, looser ones for degraded/best-effort features, so a slow ‘related suggestions’ call never blocks the main answer.

Spend the error budget. If your SLO is 99.5% success, your error budget is 0.5% — the failures you're allowed. That budget is a tool: while you're under it, ship features; if you blow it, freeze features and spend the next sprint on reliability. It turns ‘how reliable is enough?’ from an argument into a measurement — and decides when the patterns in this lesson are worth the cost. How you measure p95, error rate, and budget burn in production is the subject of L249 (Observability); this lesson is about the patterns those dashboards will be watching.

Time to feel the tradeoffs instead of reading them. The Scale & Reliability Lab is one LLM service under live load. Set the offered load (req/s) and your capacity (replicas × concurrency ÷ latency — Little's Law, live), then compose the patterns and watch the request flow respond: served vs dropped vs errored, p95 latency, GPU utilization, cost, and an SLO verdict.

Four experiments worth running:

1. Find the cliff. Drag offered load up with everything off. Below capacity it's all green; cross it and watch p95 explode and requests error out — that's can't keep up.
2. The retry trap. While overloaded, turn on Retries alone. Goodput falls and failures rise — retries amplified the load (a retry storm). Retries fix transient errors, not saturation.
3. Choose how to fail. Now turn on Load-shed: same overload, but p95 stays healthy because the excess is dropped fast (429). Compare to Queue alone, which just balloons p95 — shed beats queue for sustained overload.
4. Add real capacity. Turn on Autoscale (watch replicas climb — but remember cold starts) and a 2nd provider (an independent pool). Together they push capacity up toward demand — the redundancy story, made concrete.

The lesson the lab teaches in your hands: under overload you don't get to choose whether to fail — only how. Shed, break, fall over, or scale — but never collapse.

Predict first, then check in the lab (or in your head with Little's Law):

1. A replica handles 24 concurrent calls, each averaging 2 s. What's its throughput? How many replicas to serve 300 req/s?
2. You're at 100% of capacity and traffic ticks up 10%. With no protection, what happens to p95 and error rate? Now add load-shedding — what changes, and what's the cost to users?
3. Your provider starts returning intermittent 503s under normal load. Which tool helps most — more replicas, a circuit breaker, or a 2nd provider? Why?
4. A 15-minute research agent crashes at step 8 of 10. Without durable execution, what does recovery cost you? With it?

Answers:

1. 12 req/s per replica (24 ÷ 2). For 300 req/s you need 300 ÷ 12 = 25 replicas (then check that against provider quota — the replicas are useless if the API caps you first).
2. No protection: the 10% excess has nowhere to go — it queues, p95 spikes, then requests time out; error rate climbs far past 10% as timeouts cascade. With load-shedding: you drop ~10% fast with 429s, p95 stays flat for everyone served — you trade a little availability for bounded latency. Graceful degradation.
3. A circuit breaker helps immediately (fail fast, stop piling onto a sick endpoint, let it recover) and a 2nd provider removes the dependency entirely (failover). More replicas don't help — the problem isn't your capacity, it's their failure. (Classic 'would doubling servers fix it?' → no → reliability, not scaling.)
4. Without: you re-run from step 1 and re-pay for all 8 completed steps' tokens (and repeat any side effects). With durable execution: it replays the recorded results and resumes at step 8 — near-zero extra cost, no duplicated side effects.

• “Scaling and reliability are the same thing.” → No. Scaling is having enough capacity for the load; reliability is surviving when a dependency fails. A system can ace one and fail the other. The test: would doubling servers fix it?
• “Just autoscale on GPU utilization.” → Useless for LLMs — the GPU reads ~100% whether it's healthy or drowning. Scale on queue depth / inter-token latency.
• “Retries make everything more reliable.” → Only for transient errors, and only with backoff + jitter. During overload, retries amplify the problem — a retry storm. Use a retry budget and a circuit breaker.
• “A big queue absorbs any traffic spike.” → A queue smooths bursts. Under sustained overload it just inflates latency until everything times out. For sustained overload, shed the excess.
• “More API keys = more capacity.” → No — limits are per org/project, so extra keys share one pool. Capacity comes from independent pools: another provider, region, or provisioned throughput.
• “My provider has 99.9% uptime, so I'm fine.” → Their uptime is now your ceiling, and AI providers have routine multi-hour incidents. Without failover you inherited their outages.
• “Reliability means never dropping a request.” → Reliability means never collapsing. Serving 70% fast and rejecting 30% cleanly beats accepting 100% and dying.

• Two problems, not one. Scaling = enough capacity for the load; reliability = surviving dependency failures. Production incidents are usually the two feeding each other.

• Capacity = concurrency ÷ latency (Little's Law). The stateless tier scales by adding replicas; the model/GPU is the real bottleneck.

• Autoscale on the right signal — queue depth / inter-token latency, never CPU/GPU utilization — scale up fast, down slow, and beat cold starts by baking the model in.

• The toolkit: a timeout on every call; retry transient errors with backoff + jitter (never during overload); circuit breakers to fail fast; bulkheads to isolate; load-shed the excess (429) rather than queue it forever.

• Add real redundancy: multi-provider / multi-region failover via the gateway (L219) — and remember extra keys don't add capacity, independent pools do.

• Long agent runs need durable execution: checkpoint every step, replay to resume, idempotency keys — so a crash costs one step, not the whole (re-paid) run.

• Commit to SLOs and spend the error budget — p95/p99, tiered strict/degraded — to decide how much reliability is worth.

• Under overload you only choose how to fail — shed, break, or fall over — never collapse.

• Next — L249: Wiring Up Observability End-to-End. You now have the patterns; next you'll make them visible — metrics, structured logs, and distributed traces across the whole LLM pipeline, so you can see your p95, your error budget burn, and exactly which hop failed.