The Model Gateway & Router

In L218 we mapped the whole architecture and saw that the first thing a request hits isn't the model — it's the gateway. This lesson builds that entry layer for real: the model gateway (one secure front door to every model) and the router that decides which model each request goes to.

Here's the one-sentence version: stop letting your application call model providers directly. Put a thin proxy in front — the gateway — that presents one API and quietly handles everything that would otherwise be copy-pasted into every service: keys, rate limits, budgets, retries, fallbacks, caching, logging, and routing. In 2026 this has "graduated from a convenience to critical AI infrastructure" — it's the single control point where cost, reliability, and governance are actually enforced.

How this relates to L216. L216 taught the routing decision — given a query, which model is the cheapest one that can handle it (predict-vs-discover, RouteLLM, cascades). This lesson is about the routing (and everything else) as an architectural component — where that decision runs, what else lives alongside it, and how the app talks to it. The router is the brain; the gateway is the front door it sits behind.

In this lesson:

• Why direct calls don't scale — the mess a gateway cleans up
• What a gateway is — a unified proxy, and the pipeline a request flows through inside it
• What it centralizes — the cross-cutting concerns, in one place
• Resilience — fallback, retries, and circuit breakers (the gateway's killer feature)
• The router — choosing the model, as a component (building on L216)
• Build vs buy, and the catch: one front door is also one failure point

Scope: caching lives in the gateway but gets its own lesson (L220); guardrails (L222) and observability (L223) likewise — we'll note where they plug in and defer the depth. The routing algorithm is L216; here it's a feature of the gateway.

A prototype calls one model with one SDK and one API key. Fine. But a real app quickly needs more than one model — a cheap one for easy queries (L216), a frontier one for hard ones, maybe a self-hosted one for private data, and a backup for when a provider has an outage. The moment you have two, calling them directly from your application code rots fast:

Every problem here is a cross-cutting concern — something every call needs, that has nothing to do with your app's logic. Scatter it across services and you get key leaks, surprise bills, no failover, and vendor lock-in. The fix is the classic one from web architecture: put a proxy in front and centralize the concerns. For LLMs, that proxy is the gateway.

A model gateway (a.k.a. LLM gateway / AI gateway) is a thin proxy that sits between your application and the model providers. Your app makes one call, to one endpoint, in one format; the gateway translates it to whichever provider is configured — "a gateway presents one API to your application and translates it into calls to whichever providers you have configured" — across 250+ providers.

The same two calls from above, through a gateway:

Inside, the gateway runs every request through a pipeline — this is the mental model to hold:

request → ① budget check (reject 402 if over)
        → ② cache lookup (hit? return in <5ms)        (L220)
        → ③ ROUTE — pick the provider/model
        → ④ call the provider
        → ⑤ retry w/ backoff · circuit-breaker · FALLBACK if it fails
        → ⑥ cache the response + audit log + cost attribution   (L223)
        → response

Your application sees none of this complexity — it just sees a fast, reliable, single API. Everything in steps ①–⑥ is a concern the gateway centralizes so you write it once, not in every service.

The gateway's value is that it's the one place every cross-cutting concern lives. The big ones:

• Unified API — one request format for every provider; swap models by changing a string, not your code.
• Auth & virtual keys — your app holds a gateway key, never the providers' keys. You mint per-team / per-user virtual keys, rotate centrally, and kill a leaked key without redeploying anything.
• Rate limiting — per-user / per-model / per-route limits at the gateway, protecting upstream providers from quota exhaustion and protecting you from a runaway client.
• Cost tracking & budget enforcement — every token attributed per user/team/key; set a budget and the gateway rejects with HTTP 402 when it's hit. No more discovering spend on the monthly invoice.
• Caching — exact + semantic cache, shared across every service behind the gateway (a hit returns in <5ms vs 2–5s for a live call). (Built in L220 — it lives here.)
• Guardrails & DLP — PII scanning and content checks on the way in/out. (Built in L222.)
• Logging, tracing & governance — immutable audit logs, per-request traces, and compliance routing (zero-data-retention, data-residency) for SOC 2 / GDPR / the EU AI Act. (Observability is L223.)

The unifying principle: none of this leaks into application code. Your services stay about your product; the gateway owns the plumbing. That's the same reason web apps put an API gateway / reverse proxy in front of microservices — this is that pattern, specialized for models.

If the gateway has a killer feature, it's resilience. Model providers have outages, rate limits, and content-policy rejections constantly — and a direct call has no answer for any of them. The gateway turns each into a transparent reroute instead of a failed request:

• Retries with exponential backoff — a transient 429/5xx is retried (often up to ~5 attempts) with growing delays before giving up on a provider.
• Fallback chains — if the primary keeps failing, the request falls over to the next provider in the chain. Mature gateways distinguish three fallback categories: general (timeouts, 5xx), content-policy (one provider refused — try another), and context-window (too many tokens — route to a bigger-context model).
• Circuit breakers — when a provider+model crosses a failure threshold (e.g. >50%), the gateway trips the breaker and stops sending it traffic for a cooldown, so you're not hammering a dead provider.
• Load balancing — across healthy providers/keys via strategies like latency-based, cost-based, least-busy, or weighted (OpenRouter weights inversely to cost — a 3× pricier provider is ~9× less likely to be picked).

Configuring this is declarative — you describe the chain, the gateway executes it on every request:

This is exactly what the Gateway Lab below lets you feel: break a provider and watch the request fall over to the next healthy one — with zero change to the calling app. Without the gateway, that same outage is downtime.

Step ③ in the pipeline — ROUTE — is the router: the component that decides which model handles each request. You learned the decision in L216; here's how it lives as a component, and the three ways it commonly decides:

• Intent / task routing — a small classifier reads the query and maps it to a path: reset password → FAQ, billing → human, code → a finetuned model. (Huyen's intent classifier.)
• Semantic routing — instead of asking an LLM at runtime, pre-embed example queries for each route and route by nearest-neighbor in embedding space — fast and cheap, reusing the vector machinery from RAG.
• Cost / capability routing — the L216 lever: send easy queries to a small model, escalate hard ones — predicted up front (router) or discovered (cascade).

Where does the router live? Usually inside the gateway. A pure router decides which model; a gateway is the unified proxy that executes the call with all the cross-cutting concerns. They're conceptually distinct, but in practice most modern platforms are both — OpenRouter, Portkey, LiteLLM all expose one endpoint (gateway) with routing logic inside. So the clean mental model is:

The router decides which; the gateway handles everything else and executes. Routing is one feature of the front door — sitting right alongside auth, budgets, caching, and fallback in the same request pipeline.

Time to feel the gateway's signature move — automatic failover. Set each provider healthy / rate-limited / down, flip the gateway ON or OFF, and send requests:

The lesson in one widget: with the gateway ON, a dead primary is a non-event — the request retries and falls over to a healthy provider, and your app (talking to a single endpoint) never knows. Flip it OFF and the same outage is downtime. That asymmetry — outage → reroute vs outage → down — is why a multi-model app is a multi-provider app, and why the gateway is the entry layer.

You almost never write a gateway from scratch — the ecosystem is mature. The real choice is self-hosted vs managed:

The crossover is mostly about spend and compliance, not raw cost: at **$2k/mo** token spend a managed gateway's ~5.5$110) is far cheaper than the engineering time to self-host. Self-hosting wins when you have a data-residency / regulatory mandate, or when spend is so large the percentage fee dominates. A useful rule of thumb from the field:

• < a few $100/mo, one provider: you may not need a gateway yet — call directly, add observability.
• ~$10k–50k/mo: LiteLLM (OSS) is the common default — multi-provider, fallback, budgets, no vendor.
• > $50k/mo or governance/audit/guardrail needs: managed (Portkey, TrueFoundry) earns its keep; Kong if you're already standardized on it.

The decision rule: "if you're calling more than one model provider, or spending more than a few hundred dollars a month, the gateway stops paying for itself in convenience and starts paying for itself in money saved" — via cheaper routing, caching, and avoided downtime.

Centralizing everything into one front door is the gateway's power and its risk — they're the same fact. The moment every request flows through the gateway, the gateway is your application's availability. "When you put everything behind a single gateway, you've created the most critical failure point in your entire system. Gateway goes down, everything goes down."

So a gateway is non-negotiably a high-availability service:

• Run multiple instances across zones with a load balancer — never a single box.
• Mind its dependencies — if the gateway needs a database/cache/auth service, those are now on your critical path too; they must be HA as well.
• It adds a network hop — but the overhead is small and, in practice, "actual inference latency dwarfs gateway overhead" (a few ms vs 1–5s). Don't let micro-latency fears stop you; do keep the gateway lean (avoid one that injects hidden calls).

When not to add one (yet): a single-provider prototype spending pocket change doesn't need the extra moving part — call the provider directly and add observability first (L218's rule). Add the gateway when you cross into multiple providers, real spend, or governance requirements — which, for anything headed to production, is soon.

The trade is worth it because the gateway converts N fragile direct integrations into one hardened, observable, controllable choke point. You just have to treat that choke point with the operational respect any critical infrastructure deserves.

Reason through these, then confirm with the Gateway Lab:

1. Predict: in the lab, set the primary to 503 down and the gateway ON, then send a request. Now flip the gateway OFF and send again. Why does the same outage produce two different outcomes?
2. Your app hardcodes the OpenAI SDK + key in six microservices. A key leaks. What does a gateway change about both the blast radius and the fix?
3. Finance asks "which team spent what on LLMs last month?" With direct calls you can't answer. Which gateway feature gives you the answer, and how?
4. How is the router different from the gateway, and where does the router usually live?
5. A teammate says "a gateway is a single point of failure, so we shouldn't use one." What's the correct response?

→ (1) With the gateway ON, the request retries and falls over to the next healthy provider → success. OFF, the app calls the dead primary directly with nothing to fall over to → the request fails (downtime). The gateway is what has a fallback chain. (2) Blast radius: with a gateway your services hold a virtual key, not the provider key — the provider key lives in one place. Fix: rotate/revoke the one virtual key centrally, no redeploys; with direct calls you'd hunt the key across six services. (3) Cost tracking with per-user/team attribution — the gateway tags every token by virtual key, so spend is queryable (and you can set budgets that 402 when exceeded). (4) The router decides which model (intent/semantic/cost — L216); the gateway is the unified proxy that executes the call with auth, budgets, caching, fallback, logging. The router usually lives inside the gateway. (5) Correct that it's a critical control point — so you run it highly available (multiple instances across zones, HA dependencies). The answer to SPOF is redundancy, not avoiding the pattern; the upside (one place for cost/reliability/governance) is too large to skip.

• "A gateway is just a thin wrapper to switch models." Unifying the API is the entry ticket. The value is centralizing every cross-cutting concern — keys, budgets, rate limits, caching, fallback, logging, governance — in one place.
• "The gateway and the router are the same thing." The router decides which model; the gateway is the proxy that executes the call with everything else. They usually ship together, but they're different jobs.
• "Routing was L216, so the router is done." L216 was the decision (predict vs discover, cost/quality). L219 is the router as a component — intent/semantic/cost routing inside the gateway's pipeline.
• "A gateway adds latency, so avoid it." Its overhead is a few ms; inference latency dwarfs it, and a cache hit (which the gateway enables) is faster than any direct call. Net latency usually drops.
• "Put my provider API keys in the app and call the gateway too." No — the whole point is the app holds a virtual gateway key and the provider keys live only in the gateway. That's the security win.
• "A gateway is a single point of failure, so it's bad." It's a single control point — run it HA (multi-instance/zone). SPOF is solved by redundancy, not by scattering integrations.
• "Build the gateway in-house." Rarely needed — LiteLLM / Portkey / Kong / Cloudflare / OpenRouter are mature. Choose self-host vs managed by data-residency and spend, not by NIH.

• The gateway is one secure front door between your app and every model provider: your app makes one call, one format, one (virtual) key, and the gateway translates to 250+ providers. It's the entry layer of the architecture (L218) and, in 2026, critical infrastructure.
• It centralizes every cross-cutting concern so none leaks into app code: unified API, auth/virtual keys, rate limits, cost tracking & budgets (402), caching (L220), guardrails (L222), logging/governance (L223).
• Resilience is the killer feature — retries w/ backoff, fallback chains (general / content-policy / context-window), circuit breakers (>50% failure), and load balancing turn a provider outage into a transparent reroute instead of downtime.
• The router chooses the model (intent / semantic / cost — the L216 decision) and usually lives inside the gateway: router = which model; gateway = execute + everything else.
• Build vs buy: don't write one — LiteLLM (self-host, OSS default) vs Portkey/Cloudflare/OpenRouter (managed). Choose by data-residency & spend; the crossover favors managed at moderate scale.
• The catch: one front door is one failure point — run it highly available (multi-instance/zone). The decision rule: >1 provider or >a few hundred $/mo → you want a gateway.
• Next — L220: Adding Caches (Prompt + Semantic). The gateway is where the cache lives; next we add it — the single biggest latency-and-cost win in the whole architecture.