Fallbacks, Retries & Circuit Breakers

The guardrails so far have policed what the model says — the input (L231 — Input Guardrails: Validation, PII & Moderation), the output (L232 — Output Guardrails: Format, Safety & Grounding), and its truthfulness (L233 — Hallucination Mitigation Strategies). This lesson is about a different kind of failure: the moment the model provider doesn't respond at all — it's rate-limiting you, throwing 500s, timing out, or simply down.

The reframe that makes this whole lesson click: an LLM API call is just a network call to a flaky, rate-limited, occasionally-down third party. Strip away the magic and it's an HTTP request to someone else's server — so you defend it with the exact same battle-tested resilience patterns distributed systems have used for decades: timeouts, retries, circuit breakers, and fallbacks. None of this is AI-specific; what's specific is that LLM providers are unusually flaky (strict rate limits, heavy load, frequent incidents) and expensive (so wasted calls hurt twice).

We'll build the resilience ladder rung by rung:

• Timeout — never wait forever.
• Retry — recover from a transient blip (backoff + jitter, retryable-only).
• Circuit breaker — stop hammering a provider that's clearly down.
• Fallback — survive an outage by switching to a secondary.
• Graceful degradation — an honest partial answer beats a hard failure.

Scope: this is the infrastructure / API layer of reliability. It's distinct from Parsing, Validation & Retries (Pydantic) — re-asking the model when the output is malformed (an L232 concern) — and from Handling Tool Errors & Retries — an agent recovering from a tool failure. Here the thing that fails is the model call itself.

Engineers new to LLMs often write response = client.chat(...) as if it always returns. In production it frequently doesn't, and the reasons are mundane and constant:

• Rate limits (429). Every provider caps your requests-per-minute and tokens-per-minute. Hit the cap and you're throttled — temporarily.
• Server errors (5xx) and overload. Providers run hot. 500/503 happen; Anthropic returns 529 Overloaded under heavy load; OpenAI has its share of 500s.
• Latency spikes & hangs. A call that usually takes 2s occasionally takes 60s — or never returns.
• Real outages. Whole providers go down. This isn't rare: across major AI systems, December 2025 alone logged dozens of incidents (OpenAI and Anthropic each ~20, with hundreds of hours of cumulative impact). Single-provider dependence is now treated as a real risk — multi-provider adoption jumped from ~23% to ~40% of orgs in a year.

The consequence: your app's reliability is capped by your provider's — unless you engineer around it. If your one provider is up 99.5% of the time, your naive app is at best 99.5%. Resilience patterns are how you build a more reliable system out of less reliable parts — and the home for most of them is the model gateway from L219 (The Model Gateway & Router), the one chokepoint every call already flows through.

The cheapest, most-skipped defense. Every LLM call gets a hard timeout (a sensible default is ~30s for a non-streaming completion; tune to your p99). Why it's non-negotiable:

• A provider that hangs without a timeout holds your request open indefinitely — and with it a worker, a connection, and memory. A handful of hangs can exhaust your connection pool and take down a service that was otherwise healthy. One slow dependency becomes your outage.
• A timeout converts an unbounded hang into a bounded, catchable error — which is what lets every downstream rung (retry, breaker, fallback) actually fire.

In the lab, run the Slow / hanging scenario with the timeout off: each call ties up the request for 30s. Turn it on and the damage is capped at 8s. A timeout doesn't fix a slow provider — it contains it so the rest of your resilience stack can take over. Set it first, before anything else.

Most failures are transient — a momentary 429, a one-off 500, a brief network blip. The right response is to wait a moment and try again. But naive retry is dangerous, so there are rules:

• Exponential backoff + jitter. Wait longer after each failure (1s → 2s → 4s, capped), and add a random offset (jitter). Without jitter, every throttled client retries at the same instant — a thundering herd that re-overloads the provider the moment it recovers. Jitter de-synchronizes them.
• Honor the provider's headers. If the response includes Retry-After (or retry-after-ms), sleep exactly that long — the provider is telling you when it'll be ready. (Anthropic's 529 wants a long backoff, ~minutes.)
• Only retry what's retryable. This is the part people get wrong:
  • Retry: 429 rate-limit, 5xx/529 server errors, timeouts, connection resets — the provider's fault, and transient.
  • Never retry: 400 (malformed request), 401/403 (bad key / no access), and — the sneaky one — 429 with insufficient_quota (you're out of credits, not throttled). Retrying these just burns time and money on an error that will never succeed without a code or billing fix.
• Only retry idempotent calls. A chat completion is safe to repeat. A non-idempotent operation (e.g. submitting a fine-tune job, or any call with side effects) needs an idempotency key or it shouldn't be blindly retried.
• Cap attempts (3–5). Infinite retries don't add reliability — they add a self-inflicted DDoS on a struggling provider and exhaust your resources. After the cap, stop and escalate to a fallback.

The mental model: retry is for a blip, not an outage. If the provider is genuinely down, retrying just wastes time and money pounding a dead endpoint (watch Hard down + retry-only in the lab hit 0% uptime). That's exactly the failure the next two rungs exist to handle.

Retry assumes the provider will come back soon. But what if it's genuinely down for minutes? Then every request dutifully burns its full retry budget before failing — you're spending latency and money to pile load onto a service that can't answer, and slowing down your own users while you're at it. The circuit breaker is the pattern that says "this dependency is clearly broken — stop trying for a bit."

It's a small state machine wrapped around the provider, borrowed straight from electrical breakers and the microservices world (Hystrix → Resilience4j, Polly):

• CLOSED — normal. Requests flow; the breaker just counts failures.
• OPEN — once the failure rate crosses a threshold (e.g. >50% of the last N calls, or M consecutive failures), the breaker trips open: for a cooldown window it rejects calls instantly ("fail fast") without even trying the provider — so you don't waste time, and you give the provider room to recover.
• HALF-OPEN — after the cooldown, it lets one probe request through. If it succeeds, the breaker closes (back to normal); if it fails, it re-opens for another cooldown.

The payoff is visible in the lab: on Hard down, retry-only wastes ~9s/request and stays at 0% uptime; add the breaker and after 3 failures it trips, so later requests fail fast (~half the latency, far less cost) instead of grinding through retries. The breaker doesn't make you up — it stops the retry storm and the cascading slowdown. Pair it with the next rung to actually stay up.

Timeout contains a hang; retry rides out a blip; the breaker stops the storm — but none of them produce an answer when the primary provider is down. Fallback does: when the primary fails (or its breaker is open), route to something else. It's the single biggest lever for real uptime, and it's a chain, ordered from best to last-resort:

1. A secondary provider / model. The classic move — primary on OpenAI, fall back to Anthropic (or Azure-hosted, or a different region). A vendor's outage becomes a transparent reroute, not your downtime. This is the reason multi-provider architectures exist.
2. A cheaper / smaller model. If the flagship is overloaded, a smaller model that's up and gives a decent answer beats a perfect model that's down.
3. A cached answer. Served a similar request before? Return the cache (the semantic-cache machinery from the cost lessons doubles as a reliability fallback).
4. A canned / degraded response. The floor: "I can't reach the AI service right now — here's a help article / a human handoff."

The crucial distinction, and the one to tattoo on your brain: retry is for a blip (same provider, wait and try again); fallback is for an outage (different provider, because this one isn't coming back soon). They compose: retry the primary a couple of times → if still failing, fall back. In the lab, Hard down + fallback jumps you straight back to 100% uptime — and adding the breaker on top makes the fallback fast (later requests skip the dead primary entirely).

When the whole chain is exhausted — primary down, secondary down, no cache — you still owe the user something better than a stack trace. Graceful degradation is designing the bottom of the chain deliberately:

• Reduce scope, don't crash. Return the part you can — "I found these 3 docs but couldn't summarize them right now" — instead of failing the whole request.
• Be honest and actionable. "Our AI assistant is temporarily unavailable — here's the help center, or connect to a human." A clear, calm message with a path forward (the defensive-UX craft from L227 — Defensive UX: Designing for Uncertainty & Failure) keeps trust; a spinner that never resolves, or a raw 500, destroys it.
• Fail safe, not silent. Degradation is visible to the user and alarmed to you (it should light up the dashboards from L223 — The Monitoring & Observability Layer) — it's a handled state, not a swallowed error.

One streaming caveat (callback to L225 — Streaming UX: TTFT, Token-by-Token & Markdown Buffering): once you've started streaming tokens to the user, a mid-stream provider failure is awkward to retry or fail over — you'd have to retract what's shown. Mitigations: buffer the first chunk before committing to the stream, or fail over before the first token. Resilience and streaming UX trade off, just as they did with the output guardrails in L232.

Make the trade-offs concrete. Pick a failure mode for provider A, toggle the defenses, and fire a burst of 6 requests — watch each request's lifecycle (retries, backoff, the breaker tripping, failover) and the aggregate uptime / latency / cost:

Run the headline experiment in order on Hard down:

• Retry only → every request burns 3 retries and still fails (0% uptime) — slow and expensive. Retry can't fix an outage.
• + Circuit breaker → after 3 failures it trips; later requests fail fast. Still 0% up, but cheap and no storm. The breaker stops the bleeding.
• + Fallback → requests reroute to B → 100% uptime. Fallback is what keeps you up — and with the breaker on, the failover is fast because dead-A is skipped.

Then flip to Rate-limited (429): here retry alone wins (it's a blip), and failing over would be wasteful. Different failure, different rung — that's the whole lesson.

The rungs aren't a menu — they're a stack, applied in order on every call:

timeout → retry (the blip) → circuit breaker (the storm) → fallback chain (the outage) → degrade (the floor).

A few rules for production:

• Put it in the gateway, not every app. As with the guardrails (L222 — Putting Guardrails in the Loop), the resilience stack belongs at the model gateway (L219 — The Model Gateway & Router) — one place that owns timeouts, retry policy, breakers, and the failover chain, so every service inherits it. Managed gateways (LiteLLM, Portkey, Bedrock) ship these as config.
• Measure it. Retry rate, breaker trips, and fallback rate are leading indicators of a provider incident — wire them to the observability layer (L223) and alert on them. A silently-climbing fallback rate means your primary is degrading before it fully fails.
• Mind the cost & quality of fallbacks. A retry is a duplicate paid call; a fallback to a pricier or weaker model changes cost and quality. Resilience isn't free — but an outage costs more.
• Test it. Resilience code that's never exercised is fiction. Inject failures (chaos-style) — block a provider, force 429s — and confirm the system actually fails over. (That's what the lab lets you do safely.)

Use the Resilience Lab and what you've learned:

1. Hard down, Retry only: what's the uptime, and why is retrying a waste here?
2. Add the Circuit breaker. Uptime is still 0% — so what did the breaker actually improve, and how?
3. Now add Fallback. What happens, and what does that prove about retry vs. fallback?
4. Switch to Rate-limited (429) with Retry on. Why is retry the right tool here, and why would fallback be wasteful?
5. Slow / hanging with the Timeout off vs. on — what changes, and why is a timeout the first thing you set?

→ (1) 0% uptime — every request burns its full retry budget and still fails, costing latency and money, because retry can't revive a provider that's down (and it piles load onto it). (2) Uptime is unchanged (still down), but the breaker trips after 3 failures so later requests fail fast instead of grinding through retries — lower latency, less cost, no retry storm against the dead provider. (3) Uptime jumps to 100% as requests reroute to provider B — proving the distinction: retry handles a blip on the same provider; fallback survives an outage by switching providers. (4) A 429 is transient — wait a beat with backoff + jitter and the same provider succeeds; failing over to B would waste a healthy primary (and cost more) for a hiccup that clears in a second. (5) With the timeout off, each hang holds the request for 30s (and ties up a connection — a few of these exhaust your pool); on, it's capped at 8s. The timeout converts an unbounded hang into a bounded, catchable error, which is what lets every other rung fire — so you set it first.

• “The provider is reliable enough; I'll just call it.” Providers rate-limit, 5xx, hang, and have real outages. Your reliability is capped by theirs unless you engineer around it.
• “Just retry on any error.” Retry only retryable errors (429/5xx/timeout) — never 400/401/403 or insufficient_quota, and only idempotent calls. Retrying the rest wastes money and never succeeds.
• “Retry harder / forever to be safe.” Naive, un-jittered, uncapped retry is a self-inflicted DDoS (thundering herd) that makes an outage worse. Backoff + jitter + a cap (3–5).
• “Retry and fallback are the same thing.” Retry = a blip (same provider, wait); fallback = an outage (different provider). They compose: retry, then fall back.
• “A circuit breaker adds uptime.” It doesn't make you up — it makes you fail fast and stops hammering a dead provider. Fallback is what adds uptime; the breaker makes it efficient.
• “A timeout is optional.” A hang with no timeout can exhaust your connection pool and take down a healthy service. Set it first.
• “This is the same as Pydantic retries / tool-error handling.” Those re-ask the model on a bad output / recover a failed tool. This is resilience for the API call itself failing.

• An LLM call is a flaky network dependency — rate limits, 5xx, hangs, real outages. Defend it with classic resilience: timeout → retry → circuit breaker → fallback → degrade.
• Timeout first: cap every call (~30s) so a hang can't exhaust your connection pool — it turns an unbounded wait into a catchable error the other rungs can handle.
• Retry the blip: exponential backoff + jitter, honor Retry-After, cap at 3–5, retry only retryable errors (429/5xx/timeout — never 400/401 or insufficient_quota) and only idempotent calls. Naive retry is a self-inflicted DDoS.
• Break the storm: a circuit breaker (CLOSED → OPEN → HALF-OPEN) trips on a failure-rate threshold to fail fast and stop pounding a downed provider; a half-open probe tests recovery (Resilience4j, Polly).
• Fall back to survive an outage: a chain — secondary provider → cheaper model → cache → canned. Retry fixes a blip; fallback survives an outage; they compose. Then degrade gracefully (L227) — an honest partial beats a hard fail.
• Own it at the gateway (L219), measure it (L223), and test it: put the stack in one place, alert on retry/breaker/fallback rates as early outage signals, and inject failures to prove the failover actually works.