Routing

Introduction

Welcome to pattern 2 of 5 (L115). Prompt chaining (L116) ran the same sequence of steps on every input. Routing asks a different question first: which path should this input even take? The one-line idea, in Anthropic's words: "Routing classifies an input and directs it to a specialized followup task."

Why it matters: the beginner instinct is one giant prompt that tries to be a billing agent and technical support and a refunds desk all at once. It ends up mediocre at all three — because, as Anthropic puts it, "without this workflow, optimizing for one kind of input can hurt performance on other inputs." Routing is the fix: a small classifier out front sends each input to a specialist tuned for exactly that category.

In this lesson:

Why routing beats one do-everything prompt (separation of concerns)
Building a router from scratch — classify, then dispatch; the router never answers
The router spectrum — keyword → embedding → classifier → LLM (and routing by difficulty to cheaper models)
The two safety valves every router needs — a clarify for near-ties and a default for the unknown
How to evaluate a router (it's a classifier), the honest failure modes, when not to route, and the framework (LangGraph)

An infographic titled 'Routing — classify the input, then send it to a specialist'. One generalist prompt can't be great at everything — optimizing for one kind of input can hurt performance on other inputs — so classify the input and dispatch it to a specialist tuned for that category. THE ROUTING WORKFLOW: input (a support message) → a router that classifies it (using an LLM, an embedding/semantic router, or simple rules) → it picks exactly ONE specialist route: Billing (duplicate charges, invoices), Technical (login, crashes, how-to), or Refunds (returns, eligibility); each route is its own specialized prompt, tools, or chain. WHY ROUTE: separation of concerns — each category gets a specialized prompt instead of one bloated do-everything prompt; it works only when categories are distinct AND classification can be done accurately. THE ROUTER, cheap to rich: keyword/regex (under 1 ms), embedding/semantic router (milliseconds, one embedding call), a small fine-tuned classifier, or an LLM router (use a small/cheap model + a structured enum output); you can also route by DIFFICULTY to a cheap vs a capable model to cut cost. TWO SAFETY VALVES every router needs: a confidence threshold so a near-tie triggers a CLARIFYING question (ask, don't guess) and a low-confidence score falls back to a DEFAULT route (general assistant or human). THE HONEST FAILURE: a misroute sends the query to the wrong specialist → a confidently wrong answer, because a narrowed specialist has no generalist breadth to catch what it missed; plus a routing tax (latency/cost) and the router as a single point of failure. EVALUATE THE ROUTER AS A CLASSIFIER: measure precision, recall, and a confusion matrix per route, not aggregate accuracy; keep routes few and distinct, go hierarchical only when the fan-out hurts accuracy. Routing is not chaining (which runs every step on every input) and not the orchestrator pattern (which dynamically decomposes unknown subtasks). Takeaway: classify first, then send each input to a specialist — but only when categories are distinct and classification is accurate; add a confidence gate that clarifies a near-tie and defaults the unknown, and evaluate the router like the classifier it is.

Why Route? Separation of Concerns

The core argument is separation of concerns. A billing question and a "the app won't open" question need different knowledge, different tools, and a different tone. Cram all of that into one prompt and every instruction you add for billing slightly dilutes the model's focus on technical support — "optimizing for one kind of input can hurt performance on other inputs." Split them, and each route gets a short, sharp, specialized prompt that's easy to write, test, and improve in isolation.

Anthropic gives two canonical examples:

By category: "Directing different types of customer service queries (general questions, refund requests, technical support) into different downstream processes, prompts, and tools."
By difficulty (cost routing): route "easy/common questions to smaller, cost-efficient models like Claude Haiku 4.5 and hard/unusual questions to more capable models like Claude Sonnet 4.5." Same task, different model — also routing.

And the two preconditions — memorize these, because they're the whole "when to use":

"Routing works well for complex tasks where there are distinct categories that are better handled separately, and where classification can be handled accurately, either by an LLM or a more traditional classification model/algorithm."

Two load-bearing words: distinct (if the categories overlap, a single prompt is simpler and you just added a misclassification surface for nothing) and accurately (if you can't classify reliably, routing turns every classification error into a wrong-path answer).

Build a Router From Scratch

A router is ordinary code: one call to classify, then a second call to handle. The single most important design rule comes straight from Anthropic's cookbook — separate the routing decision from the task. The router's only job is to pick a label; a different call, with the specialized prompt, actually answers.

import anthropic
client = anthropic.Anthropic()

# Each ROUTE = a category -> its own SPECIALIZED system prompt (or tool/chain).
ROUTES = {
    "billing":   "You are a billing specialist. Resolve charge & invoice issues; you can look up transactions and issue reversals.",
    "technical": "You are a technical support agent. Diagnose login, crash, and how-to problems using the help center.",
    "refund":    "You are a refunds agent. Apply the 30-day return policy; approve or decline with the reason.",
}

def route(message: str) -> str:
    # 1) CLASSIFY — a separate call whose ONLY job is to pick a category.
    #    A small, cheap model is plenty for classification.
    label = client.messages.create(
        model="claude-haiku-4-5", max_tokens=8,
        messages=[{"role": "user",
                   "content": f"Classify this message into one of {list(ROUTES)}. "
                              f"Reply with ONLY the label.\n\n{message}"}],
    ).content[0].text.strip().lower()

    # 2) DISPATCH — run the SPECIALIZED handler for that one route.
    system = ROUTES[label]
    return client.messages.create(
        model="claude-opus-4-8", max_tokens=1024,
        system=system, messages=[{"role": "user", "content": message}],
    ).content[0].text
# The router NEVER answers the question. It only decides WHICH specialist does.

Three things make this production-grade:

The router never answers. It returns a category, not a reply. Keeping classification and handling as separate calls means you can log the decision, swap the classifier, and reuse the specialists.
Use a small, cheap model for the classifier. Picking one of three labels is easy — Haiku-class is plenty, and it keeps the routing tax (the extra call) small.
Constrain the output. In real code, force the label with a structured-output enum (Anthropic/OpenAI both support a strict schema) so the router cannot return an invalid category. The cookbook's variant has the model "first explain your reasoning, then provide your selection" in a parseable <selection> tag — reasoning improves the choice, the tag makes it machine-readable.

Notice what routing is not: it's not chaining. Chaining runs every step on every input; routing runs one path, chosen up front by classifying the input.

The Router Doesn't Have to Be an LLM

Here's a nuance most tutorials skip — and Anthropic states it explicitly: classification can be done "either by an LLM or a more traditional classification model/algorithm." The router is just a classifier, and you have a whole spectrum to choose from, cheapest to richest:

Keyword / regex — < 1 ms, deterministic, no model, fully auditable. Brittle to phrasing, but the right call for known templates and as a cheap first layer.
Embedding / semantic router — embed a few example utterances per route; at query time embed the input once and route by cosine similarity. ~milliseconds, ~$0 on a local encoder, no generation call. The semantic-router library is the canonical tool.
Small fine-tuned classifier (a BERT-class / SetFit model) — a single forward pass, and its softmax probability is the best-calibrated confidence of any option. Great for high volume + a stable set of intents.
LLM router — the most flexible (handles nuance and novel phrasings), but it's the most expensive: a full extra model call. Use a small model and constrain the output.

# The router doesn't have to be an LLM. An EMBEDDING router makes the decision
# with vector math — no generation call, ~milliseconds, ~$0 on a local encoder.
from semantic_router import Route
from semantic_router.routers import SemanticRouter
from semantic_router.encoders import HuggingFaceEncoder      # local, free, offline

routes = [
    Route(name="billing",   utterances=["charged twice", "wrong invoice", "update my card"]),
    Route(name="technical", utterances=["can't log in", "the app keeps crashing", "how do I export data"]),
    Route(name="refund",    utterances=["I want a refund", "return my order", "money back"]),
]
router = SemanticRouter(encoder=HuggingFaceEncoder(), routes=routes, auto_sync="local")

router("my card got billed twice").name     # -> 'billing'   (nearest example utterances win)
router("what's the weather today?").name     # -> None        → nothing cleared the threshold = your DEFAULT route

There's also a second flavor you'll meet constantly: model routing / cascades — route by difficulty to a cheap vs. a capable model. RouteLLM (LMSYS) reports up to ~85% cost reduction while keeping ~95% of GPT-4 quality on MT-Bench (treat that headline as benchmark-specific — the savings shrink on harder suites). A cascade (e.g. FrugalGPT) is the close cousin: call the cheap model first, and a scoring check decides whether to escalate to the expensive one. IBM reports routers can cut inference costs "by up to 85%" by sending easy queries to smaller models. Route-to-task (which specialist?) and route-to-model (how much horsepower?) are both "routing" — and you'll often do both.

The Two Safety Valves: Clarify & Default

A naive router takes the top-1 category every time — and that's exactly how routers ship embarrassing answers. Two inputs break it: an ambiguous one where two routes nearly tie, and an out-of-scope one where no route really fits. The fix is a confidence gate, and it has two valves:

# A bare argmax router CONFIDENTLY misroutes ambiguous & out-of-scope inputs.
# The fix: a confidence gate. Clarify a near-tie; default the unknown.
scores = classify_scores(message)                 # {route: confidence}
(top, p1), (second, p2) = sorted(scores.items(), key=lambda kv: -kv[1])[:2]

if p1 < 0.40:                                      # nothing is confident enough
    return default_route(message)                  #   -> general assistant or a human
if p1 - p2 < 0.12:                                 # two routes too close to call
    return ask_clarifying_question(top, second)    #   -> ASK, don't guess
return ROUTES[top](message)                        # confident -> the specialist
# Every router needs a fallback. semantic-router gives you one for free: when no
# route clears its score_threshold it returns None — that None IS your default branch.

Default route (the unknown): if the top score is below a threshold, don't force a specialist — fall back to a general assistant or a human. Every router needs a fallback. (semantic-router gives you one for free: when nothing clears its score_threshold it returns None — that None is your default branch.)
Clarify (the near-tie): if the top two scores are within a small margin, the honest move is to ask, not guess — "is this about the double charge or the crash? I'll start there."

A clean decision rule: low-confidence with one clear leader → default; two interpretations near-tied → clarify. These valves aren't in Anthropic's article — they're hard-won production practice — but they're the difference between a demo router and one you'd trust with real customers.

See It Route (and Watch a Misroute)

Pick an incoming support message below and watch the classifier's per-route confidence bars. Three messages route cleanly; two are designed to break a naive router. Then flip confidence gating off and feel the difference:

Pick an incoming support message and watch a classifier score it across three specialist routes (billing / technical / refunds) — the winner answers. Then flip CONFIDENCE GATING off: a blind top-1 router confidently misroutes the ambiguous 'charged twice AND the app crashes' message (answers the crash, ignores the money) and sends 'what's the weather?' to a billing specialist. Turn gating on and the near-tie triggers a clarifying question and the out-of-scope message falls back to a default route.

Run the two tricky messages with gating off, then on:

"I was charged twice AND the app crashes…" — gating off takes top-1 (technical, because "crashes" dominates) and walks the customer through clearing their cache — never addressing the double charge, the thing that actually matters. A narrowed specialist has no breadth to notice what it missed. Gating on sees the near-tie and asks which to tackle first.
"What's the weather today?" — gating off still picks a top-1 (billing, absurdly). Gating on sees the max score is below the bar and routes to a default handler.

That's the whole lesson in motion: a misroute is a confidently wrong answer, and the confidence gate (clarify + default) is what prevents it.

Evaluate the Router Like the Classifier It Is

This is the move that separates a senior engineer from a beginner: your router is a classifier, so evaluate it like one. A single end-to-end "was the answer good?" score hides where it broke and is misleading when your route distribution is imbalanced.

Instead, build a labeled set of inputs → correct routes and measure the router on its own:

Precision / recall / F1 per route, and a confusion matrix — which categories get mistaken for which? (Billing↔technical confusion is the classic; the "charged twice AND crashes" message lands right there.)
Monitor the live route distribution. A swelling default bucket is your early warning that new intents have appeared that no route covers ("stale routes") — time to add or split a route.
Watch for too many routes. Classifier accuracy degrades as the fan-out grows; if you have many categories, go hierarchical (route to a coarse group, then a sub-router) so no single classifier faces every destination at once.

The Honest Tradeoff (and When NOT to Route)

Routing earns its place, but a world-class engineer knows its costs:

The routing tax. Every request pays for the classification step (an extra call's latency + cost for an LLM router). It must be cheaper than the specialization is worth — which is exactly why embedding/keyword routers exist.
The router is a single point of failure. Everything flows through it; if it's down or degraded, the whole system is. (Deploy it redundantly.)
A misroute is worse than a generalist. A mis-targeted specialist answers confidently in the wrong frame; a generalist would at least have attempted the real question. This is why accurate classification is a precondition, not a nice-to-have.

When to route: inputs fall into distinct categories that a single prompt genuinely can't serve well, and you can classify them accurately — customer-service triage, doc-type-specific extraction, cheap-vs-capable model selection.

When NOT to route: one generalist prompt already handles everything fine (start there — add routing only when cost/latency/quality forces it); the categories aren't actually distinct; you can't classify accurately; or it's a 2-case problem where a single prompt with a conditional instruction beats a whole router + classifier eval + fallback machinery.

And keep the family straight: Routing picks one of N predefined paths up front. Chaining (L116) runs a fixed sequence on everything. Orchestrator-Workers (L119) dynamically decomposes subtasks it couldn't enumerate in advance. Routing's routes are known and fixed; an orchestrator's subtasks are not.

Then — and Only Then — the Framework

You just built a router in plain Python, so a framework will never be a black box. In LangGraph, routing is a first-class conditional edge — a pure function reads the state and returns the name of the next node:

# First principles first (above) — THEN a framework. In LangGraph, routing is a
# first-class CONDITIONAL EDGE: a pure function reads state and returns a node name.
def pick_route(state) -> str:
    return classify(state["message"])              # "billing" | "technical" | "refund" | "default"

graph.add_conditional_edges("router", pick_route, {
    "billing":   "billing_node",
    "technical": "technical_node",
    "refund":    "refund_node",
    "default":   "general_node",                   # map EVERY case — there's no implicit "else"
})
# LangChain's old LLMRouterChain / MultiPromptChain are DEPRECATED; conditional
# edges (or a RunnableBranch / RunnableLambda) are the modern way to route.

add_conditional_edges(source, fn, path_map) is the modern idiom. Note one sharp detail: there's no implicit "else" — your function must return a valid node name for every input, so you implement the default route by returning your catch-all node's name on low confidence.

Two more landmarks: LangChain's old LLMRouterChain / MultiPromptChain are deprecated (the ecosystem moved to Runnables / LangGraph — use a RunnableBranch or conditional edges instead); and the semantic-router library is the go-to for the embedding approach. A related idea you'll meet in the multi-agent part (L145): the OpenAI Agents SDK treats a handoff as routing where the chosen specialist takes over the conversation — same classify-and-dispatch, but ownership transfers. Build the router once by hand; the framework is just sugar over the classify-then-dispatch you already understand. (Same lesson as L111 — Building a ReAct Agent From Scratch and L116 — Prompt Chaining.)

🧪 Try It Yourself

Reason through these, then use the widget to confirm:

Predict before toggling: set the "charged twice AND the app crashes" message and turn gating off. Which route wins, what does the customer get, and what's the name for what just happened?
You're routing incoming emails to sales / support / careers / spam. Your router is an LLM call and your volume is 2M emails/day. What's the first change you'd make, and why?
Your support router's default bucket has quietly grown from 3% to 22% of traffic over a month. What does that almost certainly mean, and what do you do?
A teammate reports the router is "85% accurate" and wants to ship. Why is that number not enough, and what would you ask to see instead?
When would you collapse a 3-route router back into a single prompt — i.e., when is routing over-engineering?

→ (1) Technical wins (the word "crashes" dominates the signal); the customer gets cache-clearing steps and the double charge is never addressed — a misroute, i.e. a confidently wrong answer from a narrowed specialist. (2) Swap the LLM router for a cheaper classifier — an embedding/semantic router or a small fine-tuned model — and/or a keyword first layer. At 2M/day the per-request routing tax of an LLM call dominates, and an embedding router is ~milliseconds and ~free while being plenty accurate for a few distinct buckets. (3) New intents have appeared that no route covers (stale routes) — too many inputs are failing the confidence bar and falling to default. Inspect the default bucket, cluster the misses, and add/split a route (then re-evaluate). (4) Aggregate accuracy hides where it breaks and is misleading on imbalanced traffic. Ask for per-route precision/recall and a confusion matrix — a router that nails the 90%-common route but mangles refunds can still read "85%." (5) When the categories aren't really distinct (one prompt serves them all), classification isn't reliable, or the routing tax exceeds the specialization benefit — then the extra classifier + fallback machinery is pure overhead.

Mental-Model Corrections

"Routing is just chaining with an if-statement." No. Chaining runs the same steps on every input; routing classifies up front and runs one of N distinct paths. Different shape, different purpose.
"The router should also answer the question." Keep them separate. The router returns a category; a different call (the specialist) answers. Mixing them makes both worse and un-loggable.
"A router has to be an LLM." Often the worst choice on cost/latency. A keyword, embedding, or small classifier router is frequently more accurate-per-dollar — Anthropic explicitly endorses "a more traditional classification model/algorithm."
"Take the top category, always." That's how you ship confident misroutes. Add a confidence gate: default the low-confidence, clarify the near-tie.
"A misroute is no big deal — close enough." A mis-targeted specialist is confidently wrong with no breadth to recover. That's often worse than a generalist's honest attempt.
"We measured the router — 85% accurate, ship it." Measure it as a classifier: per-route precision/recall + a confusion matrix, not one aggregate number.
"I need LangChain to route." You need a classifier call and a dict. LangGraph's conditional edge is just that, made graph-native (and LLMRouterChain is deprecated anyway).

Key Takeaways

Routing = classify an input, then direct it to a specialized followup task. Pattern 2 of 5. It buys separation of concerns — a sharp specialized prompt per category instead of one bloated do-everything prompt.
Two preconditions (Anthropic): the categories are distinct and classification can be done accurately (by an LLM or a traditional classifier).
Build it from scratch: classify → dispatch. The router returns a category, never the answer; use a small model + a structured enum, and keep the routes as a dict of specialized prompts.
The router spectrum: keyword (<1 ms) → embedding/semantic (~ms, $0 local) → fine-tuned classifier (best-calibrated) → LLM (flexible, priciest). Plus model routing/cascades to send easy queries to cheap models (RouteLLM: up to ~85% cost cut on MT-Bench).
Two safety valves: a default route for low confidence (every router needs a fallback) and a clarify for near-ties (ask, don't guess).
Evaluate it like a classifier: per-route precision/recall + a confusion matrix; watch the default bucket for stale routes; go hierarchical if too many routes hurt accuracy.
Honest costs: a routing tax, a single point of failure, and misroutes that are confidently wrong. Don't route a non-distinct or 2-case problem.
Next: Parallelization (L118) — when the right move isn't to pick one path, but to run several at once and aggregate.