Model Routing & Cascades (Heterogeneous Architectures, SLMs)

Introduction

The last two lessons cut cost by not recomputing: prompt caching reused the prefix (L214), semantic caching reused the answer (L215). This one cuts cost a different way — by not over-paying for the model itself.

Here's the waste most teams ship with: one big model answers everything. But your traffic isn't uniform. "Classify this as spam or not" and "prove this theorem and write the production code" are wildly different jobs — and from L213 you know the model tier swings cost ~5×. Sending the easy query to a frontier model is like hiring a senior architect to reset a password. It works, but you're burning money on every easy request — which, for most products, is the majority of requests.

The core move: match the model to the query. Send each query to the cheapest model that can handle it — a small model for the easy bulk, a frontier model only when the work genuinely needs it. Do that well and you get ~frontier quality at a fraction of the cost.

There are two ways to decide, and this lesson is built around the contrast:

Routing — a small, fast classifier predicts difficulty up front and dispatches in one shot.
Cascades — try the cheap model first; if the answer isn't good enough, escalate to a stronger one.

We'll also cover the agent angle the title calls out — heterogeneous architectures and small language models (SLMs) — the real-world numbers (RouteLLM, FrugalGPT), and the failure mode that makes this dangerous: a misroute fails silently, returning a confident wrong answer.

Scope: this is about choosing the model per request. It pairs with caching (L214–L215) — route the calls you can't cache. Next (L217) is the last cost lever: batch APIs & distillation.

Infographic titled 'Model Routing & Cascades'. The big idea: don't send every query to your most expensive model — send each query to the CHEAPEST model that can handle it. CENTER, two strategies side by side. STRATEGY 1, the ROUTER: an incoming query hits a small, fast CLASSIFIER that predicts difficulty up front and dispatches it in one shot — easy queries go to a small model (Haiku-class), medium to a mid model (Sonnet-class), hard to a frontier model (Opus-class). One call, plus about 10 to 50 milliseconds of routing overhead; the risk is a MISROUTE — a hard query sent to a small model returns a confident WRONG answer. STRATEGY 2, the CASCADE: try the cheap model FIRST, then a scorer asks 'is this good enough?' — if yes, return it; if no, ESCALATE to a stronger model. No classifier needed and it's robust, but escalated queries pay twice and take about 2x latency (two sequential calls). A spectrum strip shows the trade: ALL-CHEAP is cheapest but gets the hard queries WRONG (the trap — cheap is worthless if it's wrong, and it fails silently); ALL-FRONTIER is 100 percent correct but overpays on every easy query; ROUTING and CASCADES sit in the sweet spot — about frontier quality at a fraction of the cost. THREE CARDS. Card 1, heterogeneous architectures and SLMs: most agent steps are narrow and repetitive (classify, extract, format, simple tool calls) and don't need a frontier model — small language models (Phi, Gemma, Qwen, Llama-small, Haiku-class) are about 10 to 30 times cheaper and faster, so compose a system that uses small models for the bulk and a frontier model only for the hard reasoning. Card 2, the real numbers: RouteLLM reports up to about 85 percent cost reduction while keeping about 95 percent of GPT-4 quality; FrugalGPT-style cascades match or beat a frontier model at roughly a quarter of the cost. Card 3, the dangers and when NOT to route: silent degradation from misroutes (you must eval continuously), the router goes stale when models change (retrain it), and below roughly ten thousand requests a day the engineering overhead outweighs the savings — routing is an optimization, not a foundation. Section roadmap strip: token economics, prompt caching, semantic caching, model routing, batch and distillation. Takeaway banner: match the model to the query — route by predicting difficulty up front, or cascade by trying cheap and escalating — to hit near-frontier quality at a fraction of the cost, but eval continuously because a misroute fails silently.

The Core Idea — Don't Pay Frontier Prices for Easy Work

Start with the economics from L213: across a model family, the price per token swings roughly 5× from the small tier to the frontier tier, and latency swings with it. Now look at a realistic production query mix:

Query	Real difficulty	Cheapest model that nails it
"Classify sentiment"	trivial	small
"Extract the invoice date"	trivial	small
"Summarize this ticket"	moderate	mid
"Debug this race condition"	hard	frontier

If one frontier model answers all four, three of them are overpaying. The whole field of routing exists to fix exactly this. Two numbers frame the opportunity and the risk:

The opportunity: in most production traffic, the majority of queries are easy — and easy queries are where a small model is indistinguishable from a frontier model. That's free money.
The risk: the queries a small model gets wrong are exactly the ones that matter most (the hard, high-stakes ones). Misjudge difficulty and you ship a confident wrong answer.

So routing is fundamentally a bet on each query's difficulty, and the entire engineering problem is: how accurately, and how cheaply, can you make that bet? The two strategies answer that question differently — one predicts difficulty, the other discovers it.

Two Strategies — Routers (Predict) vs Cascades (Discover)

Strategy 1 — the Router (predict up front). A small, fast classifier looks at the query before any expensive model runs and predicts which tier it needs, then dispatches in one shot.

query ─▶ [router/classifier] ─▶ small model   (easy)
                              ├▶ mid model     (medium)
                              └▶ frontier model (hard)

Cost/latency: one model call + ~10-50ms of routing overhead. Cheap and fast when the router is right.
The failure mode: a misroute. Send a hard query to the small model and you get a fluent wrong answer — and the router never finds out it was wrong.
How the router decides: a few flavors — a BERT-style classifier or matrix-factorization / similarity-weighted model trained on which queries each model handles well (RouteLLM); an embedding / semantic router that matches the query to labeled clusters; or an LLM-as-router (ask a cheap model "which tier?"). All of them add a small, separate model you now have to train and keep fresh.

Strategy 2 — the Cascade (discover by trying). Don't predict — try the cheap model first, then score its answer. Good enough? Return it. Not good enough? Escalate to a stronger model.

query ─▶ [cheap model] ─▶ [good enough? scorer] ─▶ yes ─▶ return
                                                  └▶ no  ─▶ [stronger model] ─▶ return

Cost/latency: queries the cheap model handles are cheap; escalated queries pay twice and take ~2× latency (two sequential calls). Net win when the cheap model clears a large fraction on its own.
The hard part is the scorer — "is this answer good enough?" A cheap self-evaluation, a verifier model, a confidence/logprob check, or a task-specific test (does the code run? is the JSON valid?). The scorer is the cascade's brain; a bad scorer either escalates everything (no savings) or accepts garbage (wrong answers).
The upside: no upfront classifier to train, and it's robust — the cheap model's mistakes get caught and fixed by escalation, so quality stays high.

Here's the trade in one line: a router decides once and cheaply but can be wrong; a cascade is self-correcting but pays extra latency/cost on the queries it escalates. Many production systems combine them — a router for the obvious cases, a cascade for the uncertain ones.

# Strategy 1 — ROUTER: a small classifier predicts difficulty, then one dispatch.
SMALL, MID, FRONTIER = "claude-haiku-4-5", "claude-sonnet-4-6", "claude-opus-4-8"

def route(query: str) -> str:
    tier = difficulty_classifier(query)        # BERT/MF/embedding/LLM-as-router → "easy" | "med" | "hard"
    model = {"easy": SMALL, "med": MID, "hard": FRONTIER}[tier]
    return call(model, query)                   # ONE call. Risk: a misroute returns a confident wrong answer.

# Strategy 2 — CASCADE: try cheap, score, escalate only if needed.
def cascade(query: str) -> str:
    ans = call(SMALL, query)                    # 1) try the cheap model
    if good_enough(query, ans):                 # 2) the scorer is the whole game
        return ans                              #    HIT — cheap and done
    return call(FRONTIER, query)                # 3) MISS — pay again, ~2x latency, but correct

# `good_enough` options: self-eval ("rate your confidence 0-1"), a verifier model,
# a logprob/confidence threshold, or a hard check (code runs? schema valid? answer parses?).

See It — The Routing Lab

Eight queries — four easy, two medium, two hard — sent through all four strategies. Flip between them and watch cost and quality move together:

One batch, eight queries — four easy, two medium, two hard — sent four ways. **All-frontier** sends every query to the big model: 100% correct, but $0.10 (the baseline). **All-cheap** sends everything to the small model: $0.02 (80% cheaper) but the medium and hard queries come back **wrong** — 50% quality, and it fails *silently*. **Router** classifies each query and sends it to the cheapest *capable* tier: 100% quality at **$0.05 — half the cost** of all-frontier. **Cascade** tries the cheap model first and escalates the ones it can't handle: also 100% quality, at $0.07 — more than the router (it pays for the cheap miss, then jumps to the strong model) but with **no classifier to train.** Flip through the four and watch cost and quality move together. Illustrative costs.

What the lab makes concrete:

All-frontier is the quality baseline: 100% correct, but $0.10 — it pays frontier price for six queries that didn't need it.
All-cheap is the trap: cheapest at $0.02 (80% off), but the medium and hard queries come back wrong → 50% quality. And nothing errors — it fails silently, so you only find out from users churning or a degraded eval.
Router is the sweet spot: by sending each query to the cheapest capable tier, it hits 100% quality at $0.05 — half the cost of all-frontier.
Cascade also reaches 100% quality at ** $0.07** — *more* than the router, because it pays for the cheap attempt and then jumps to the strong model — but it needs **no classifier.** That gap ($ 0.07 vs $0.05) is exactly the price of discovering difficulty instead of predicting it.

The ranking — cheap < router < cascade < frontier on cost, with only all-cheap losing quality — is the whole lesson in four bars.

Heterogeneous Architectures & SLMs — The Agent Angle

Routing a chatbot's traffic is the simple case. The bigger opportunity — and the reason the title says "heterogeneous architectures, SLMs" — is agents.

From L213, agents are the cost bomb: a single user request fans out into dozens of model calls around the agent loop. But here's the key observation (made forcefully in NVIDIA's 2025 "Small Language Models are the Future of Agentic AI"): most of those calls are narrow and repetitive — parse this tool output, decide the next step from a fixed menu, extract three fields, format as JSON, classify intent. They do not need a frontier model.

That motivates a heterogeneous architecture: instead of one big model driving every step, you compose a system where small models do the bulk of the narrow work and a frontier model is reserved for the genuinely hard reasoning (the plan, the tricky judgment call). Routing is what dispatches each step to the right model.

Small Language Models (SLMs) are what make the bulk cheap. Roughly ≤10B parameters, they're ~10-30× cheaper and faster than frontier models and often run on-device or self-hosted:

Open weights: Phi-3.5-mini (3.8B), Gemma-2 (2B/9B), Qwen2.5 (0.5-7B), Llama-3.2 (1B/3B).
API "small" tiers: Claude Haiku, GPT-class mini, Gemini Flash.

Where SLMs shine: classification, extraction, formatting, routing itself, simple/structured tool calls, and fixed-schema generation — exactly the high-volume, low-ambiguity steps that dominate an agent's call count. Where they don't: open-ended multi-step reasoning, long-context synthesis, hard code/math. A well-architected agent therefore looks less like "GPT-4 in a loop" and more like "a fleet of small models with a frontier model on call."

Why this is the highest-leverage place to route: the savings compound over every call in every loop, on every request. Shaving a 50-step agent from all-frontier to 80%-SLM doesn't just cut the bill — it cuts latency per step too, which is what makes long agent runs tolerable.

The Numbers — What Real Systems Achieve

Routing isn't a marginal optimization — the published results are dramatic, which is why it's a standard production lever:

RouteLLM (LMSYS, 2024) — an open-source router framework trained on Chatbot Arena preference data. On MT-Bench it reached up to ~85% cost reduction while preserving ~95% of GPT-4's quality; on harder benchmarks (MMLU, GSM8K) it cut cost ~35-46% at the same ~95% quality bar. Two findings matter for you: the routers are data-efficient (a useful one trains on <1,500 samples) and they transfer — a router trained on one model pair generalizes to others without retraining.
FrugalGPT (Stanford, 2023) — the paper that popularized the LLM cascade. By querying models cheap→expensive and stopping when a reliability scorer accepts, it matched or beat the best single model's accuracy at roughly a quarter of the cost (and far more on easy-heavy workloads). It pairs the cascade with two siblings you already know: prompt adaptation and LLM approximation (≈ caching, L215).
SLM substitution — replacing frontier calls with SLMs on the narrow agent steps they handle yields the ~10-30× per-call savings above, compounded across the loop.

The pattern across all of these: you can hold quality roughly constant (~95-100% of frontier) while cutting cost by roughly half to roughly 4× — if your traffic has enough easy queries (it usually does) and if your router/scorer is good. That "if" is the engineering, and the next section is where it goes wrong.

Build vs buy: you rarely write a router from scratch. LLM gateways do it for you — OpenRouter, LiteLLM, Portkey, Martian, Unify, Not Diamond, and RouteLLM itself — handling routing, fallback, and multi-provider access behind one endpoint. Reach for one before hand-rolling.

The Dangers & When NOT to Route

Routing adds a new decision to your system, and every decision can be wrong. The failure modes:

Silent degradation (the big one). A misroute — or a too-eager cascade scorer — returns a fluent, confident, wrong answer. There's no error, no exception. The only defense is continuous evaluation: sample routed traffic, check the answer was actually right, and track quality per tier. A router you don't eval is a quality regression you haven't noticed yet.
Router obsolescence. Your router learned which queries the current models handle well. Models change — you upgrade the small model, a provider ships a new version — and the router's assumptions go stale, misrouting in new ways. Routers need retraining/re-tuning on a schedule, not set-and-forget.
A new failure point & added complexity. The router/gateway is now on your critical path. It can be down, slow, or buggy. You've traded model cost for system complexity — make sure the savings justify it.
Cascade latency tax. Every escalated query is 2× (or more) the latency of going straight to the strong model. If escalations are common, the cascade can be slower and barely cheaper — tune the scorer so the cheap model clears a real majority.

When NOT to route:

Low volume. Below roughly 10,000 requests/day, the engineering + eval overhead usually outweighs the savings. Just use one good model.
Uniform difficulty. If nearly every query is hard (or all trivial), there's nothing to route — pick the one right model.
Before you've shipped. Routing is an optimization, not a foundation. Get a single capable model working and measured first; add routing once you have real traffic, real costs, and an eval to protect quality. Premature routing optimizes a system you don't understand yet.

The discipline: treat routing like any other optimization — measure first, protect quality with an eval, and only ship it where the volume makes the savings worth the complexity. The win is real and large; so is the silent-failure risk if you skip the eval.

🧪 Try It Yourself

Reason through these, then confirm with the lab:

Predict: in the lab, why does all-cheap cost the least but still lose to the router? What exactly does the router buy you for its extra $0.03?
Why does the cascade ( $0.07)** cost *more* than the **router ($ 0.05) even though both reach 100% quality? Where does the extra money go?
You run a support bot doing 2,000 requests/day. Should you add a router? Why or why not?
Your agent makes ~40 calls per user task; ~32 are "extract a field / pick the next action / format output." Which lesson concept attacks this, and roughly how much per-call savings is on the table?
You ship a router, costs drop 50%, everyone's happy. Three weeks later your small-model provider pushes a new version. What can silently break, and what process should have caught it?

→ (1) All-cheap is cheapest because it never escalates — but it answers the medium and hard queries wrong (50% quality), silently. The router's extra $0.03 buys **correctness on those queries** by sending them to a capable tier — cheap is worthless if it's wrong. **(2)** The cascade pays for the **wasted cheap attempt** on every hard query *and then* escalates to the strong model (and a medium query it can't clear gets escalated all the way up). The router skips the wasted attempt by **predicting** the tier directly — that's the$ 0.02 gap, the price of discovering difficulty vs predicting it. (3) Probably not — at 2k/day you're below the ~10k/day rule of thumb; the eval + maintenance overhead likely exceeds the savings. Use one good model. (4) Heterogeneous architecture + SLMs: those 32 narrow steps are SLM-shaped, and an SLM is ~10-30× cheaper per call — compounded across the loop, that's the bulk of the agent's bill. (5) A new small-model version changes which queries it handles well, so your router misroutes in new ways and quality silently degrades. Continuous evaluation (sampling routed traffic, checking correctness per tier) plus scheduled router re-tuning should have caught it.

Mental-Model Corrections

"Always use the best model — it's safest." It's the most expensive, and on easy queries a small model is indistinguishable. All-frontier overpays on the majority of traffic.
"Cheaper is the goal, so route everything to the small model." That's all-cheap — the trap. It's cheapest and wrong on the hard queries, and it fails silently. Cheap is worthless if it's wrong.
"Routing and cascading are the same thing." A router predicts difficulty up front (one call, can misroute). A cascade discovers it by trying cheap and escalating (self-correcting, but pays extra latency/cost on escalations).
"A router can't be wrong — it's just a classifier." A misroute returns a confident wrong answer with no error. Silent degradation is the #1 risk; you must eval continuously.
"Set up the router once and forget it." Models change; the router goes stale and misroutes in new ways. Retrain/re-tune on a schedule.
"SLMs are just worse models." For narrow, repetitive work (classify, extract, format, simple tools) they're as good and ~10-30× cheaper — which is most of an agent's calls. Frontier models earn their price only on the hard reasoning.
"Route from day one." Routing is an optimization, not a foundation. Ship one good model, measure, then route at volume (≳10k req/day) with an eval guarding quality.
"I'll build the router myself." Usually unnecessary — LLM gateways (OpenRouter, LiteLLM, Portkey, Martian, RouteLLM…) do routing, fallback, and multi-provider access for you.

Key Takeaways

Match the model to the query. The tier swings cost ~5× (L213), so sending easy queries to a frontier model overpays on the majority of traffic. Routing sends each query to the cheapest model that can handle it → ~frontier quality at a fraction of the cost.
Two strategies. Router = a classifier predicts difficulty up front (one call + ~10-50ms; risk: a misroute → confident wrong answer). Cascade = try cheap, score, escalate (no classifier, self-correcting; cost: escalated queries pay twice + ~2× latency).
The lab's ranking: cost is cheap < router < cascade < frontier, and only all-cheap loses quality. The router's edge over the cascade is the price of predicting vs discovering difficulty.
Heterogeneous architectures & SLMs are the agent payoff. Most agent steps are narrow/repetitive → use SLMs (Phi, Gemma, Qwen, Llama-small, Haiku-class; ~10-30× cheaper) for the bulk and a frontier model only for hard reasoning. Savings compound over every call in every loop.
The numbers are real: RouteLLM ~85% cheaper at ~95% GPT-4 quality; FrugalGPT cascades match a frontier model at ~¼ the cost. Don't build it yourself — LLM gateways (OpenRouter, LiteLLM, RouteLLM…) route for you.
The danger is silent degradation. A misroute returns a confident wrong answer with no error → eval continuously (per-tier correctness) and re-tune the router when models change.
When NOT to route: below ~10k req/day, uniform difficulty, or before you've shipped. Routing is an optimization, not a foundation — measure first, protect quality with an eval, then route at volume.
Next — L217: Batch APIs & Distillation for Cost — the last cost lever: trade latency for a ~50% discount on non-urgent work, and distill a frontier model's behavior into a small one you own.