Orchestration & Pipelines

Introduction

You've built the components — the gateway (L219), the cache (L220), and you know the model, retrieval, and guardrails pieces. But components in a drawer aren't an application. Orchestration is the layer that wires them into one end-to-end pipeline — the glue that turns "a query enters" into "a response leaves," through every step in between.

From L218 you know orchestration is one of the two cross-cutting layers (with observability) — it doesn't sit at one point in the flow; it defines the flow. This lesson makes that concrete: how a pipeline is actually assembled, scheduled, and made reliable — and the one decision every team faces: use a framework, or write it in plain code?

The headline skill is latency: a pipeline is a graph, not a line, and the biggest, cheapest win in the whole architecture is running the independent steps in parallel. Get the schedule right and the same work finishes 30–40% faster — for free.

In this lesson:

What orchestration is — define components, then chain them (Huyen's two stages)
The pipeline is a graph — sequential, parallel, branching, loops
Parallelize for latency — wall-clock = the critical path, not the sum
Error handling & durability — retries, fallback, circuit breakers, resumable workflows
Framework or plain code — LangChain / LangGraph / LlamaIndex / Haystack / DSPy, and "start without one."

Scope: the five agent workflow patterns (prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer) are the Agents material — here we focus on the orchestration layer of the application architecture. Routing is L216/L219, caching L220, guardrails L222, observability L223; we wire them together, and defer each one's depth to its lesson.

Infographic titled 'Orchestration & Pipelines' — the glue layer of a production LLM application. The big idea: orchestration chains the components (gateway, guardrails, router, retrieval, cache, model) into ONE end-to-end pipeline, and that pipeline is a GRAPH (a DAG), not a straight line. THE TWO STAGES, from Chip Huyen: first, COMPONENTS DEFINITION — tell the orchestrator what models, databases, tools, and evaluators are available; second, CHAINING / PIPELINING — sequence the steps and pass data between them, with branching, parallelism, loops, and error handling. THE BIGGEST LATENCY WIN: parallelize. A naive SEQUENTIAL pipeline runs steps one after another, so wall-clock time is the SUM of every step. But independent steps — input guardrail, router, and several retrievals — don't depend on each other, so an ORCHESTRATED pipeline runs them CONCURRENTLY, and wall-clock drops to the CRITICAL PATH: the longest chain of steps you cannot parallelize (the slowest retrieval, then build prompt, then the model call, then the output guardrail). Same work, same cost — a smarter schedule, often 30 to 40 percent faster. But you cannot parallelize past a real dependency: the model needs the retrieved context first, and the output guardrail needs the model's answer. ERROR HANDLING lives in the pipeline too: retries with exponential backoff, fallbacks (from the gateway, L219), circuit breakers, and conditional branching so a downstream step reacts to an upstream failure; long-running or agentic pipelines add durable execution (checkpoint and resume). FRAMEWORK OR PLAIN CODE: LangChain is the general-purpose orchestrator; LangGraph adds stateful, graph-based execution with cycles and human-in-the-loop for complex and agentic flows; LlamaIndex specializes in retrieval; Haystack builds typed, visualizable DAG pipelines; DSPy compiles and optimizes prompts programmatically. But the canonical advice from Chip Huyen is to START WITHOUT A FRAMEWORK — build the pipeline in plain code first, and only adopt a framework when its integration, complex-pipeline support, and ease of use clearly pay for themselves, avoiding any tool that hides API calls or injects latency. Roadmap strip: reference architecture (L218), gateway and router (L219), adding caches (L220), orchestration (L221), guardrails (L222), observability (L223). Takeaway banner: orchestration is the glue that chains components into one end-to-end pipeline that is a graph, not a line — its biggest win is parallelizing the independent branches so wall-clock becomes the critical path, not the sum, and the discipline is to start in plain code and add a framework only when it earns its keep.

What Orchestration Is — Define, Then Chain

Strip away the frameworks and orchestration is two stages (Chip Huyen's framing):

Components definition — tell the orchestrator what exists: which models, retrievers, databases, tools, caches, guardrails, and evaluators it can call.
Chaining (pipelining) — sequence those components into the end-to-end flow, passing data from each step to the next.

The "chain" is the request lifecycle from L218, made executable. A typical pipeline:

process query → retrieve context → build prompt → call model → evaluate output → return (or escalate)

That's the orchestrator's job: hold that sequence, run each step, hand the output of one to the input of the next, and decide what happens on success or failure. Everything else in this lesson — parallelism, branching, error handling, frameworks — is about doing this well: faster, more reliably, and without drowning in glue code.

The deepest point first: orchestration is where your architecture stops being a diagram and becomes a program. The diagram (L218) says what connects to what; the orchestrator says in what order, with what data, and what to do when a step fails. That's a real engineering surface — and the rest of the lesson is its core decisions.

The Pipeline Is a Graph, Not a Line

It's tempting to picture the pipeline as a straight line — step 1 → step 2 → step 3. Real pipelines are graphs (specifically DAGs — directed acyclic graphs, plus loops for agents). Four shapes show up constantly:

Sequential (A → B → C) — each step needs the previous one's output. Simplest, most debuggable; wall-clock = the sum of the steps.
Parallel (A, B, C at once → merge) — independent steps run concurrently and a downstream step merges them; wall-clock = the slowest branch, not the sum. (The next section.)
Branching / conditional (if X → B, else → C) — the path depends on a value: the router picks a model, a guardrail decides pass-or-retry, an intent classifier forks the flow.
Loops (… → check → not done? → back) — iteration and the agent loop from L218 (feed the output back as the next input).

Why does the shape matter? Because the shape determines the latency, the failure modes, and the complexity. A sequential chain is easy to reason about but slow; a parallel fan-out is fast but needs a merge step and careful error handling; branches and loops add power but make the flow harder to test. Good orchestration is choosing the right shape for each part of the pipeline — and the highest-leverage choice is where to parallelize.

Parallelize for Latency — Wall-Clock Is the Critical Path

Here's the single most valuable orchestration move. Look at the start of a real request: before the model can run, you often need to scrub PII (input guardrail), classify intent (router), and retrieve context (vector + web + SQL). None of these depend on each other — yet a naive pipeline runs them one after another, so they pay their sum.

Run them concurrently and the fan-out costs only its slowest branch. The wall-clock of the whole pipeline becomes the critical path — the longest chain of steps you can't parallelize:

import asyncio

# ✗ SEQUENTIAL — wall-clock = SUM of every step (≈ 2.45s here), most of it spent idle
async def slow(query, ctx):
    safe   = await input_guardrail(query)     # 250ms ─┐
    intent = await route(query)               # 120ms  │ none of these four
    vec    = await retrieve_vector(query)      # 200ms  │ depend on each other —
    web    = await retrieve_web(query)         # 450ms  │ but we wait for each
    sql    = await retrieve_sql(query)         # 350ms ─┘ in turn (1370ms idle!)
    prompt = build_prompt(query, vec, web, sql)
    out    = await model(prompt)              # 800ms
    return await output_guardrail(out)        # 250ms

# ✓ ORCHESTRATED — fan out the INDEPENDENT steps; wall-clock = the CRITICAL PATH (≈ 1.53s, −38%)
async def fast(query, ctx):
    safe, intent, vec, web, sql = await asyncio.gather(   # all at once → costs the SLOWEST (web, 450ms)
        input_guardrail(query), route(query),
        retrieve_vector(query), retrieve_web(query), retrieve_sql(query),
    )
    prompt = build_prompt(query, vec, web, sql)
    out    = await model(prompt)              # ← can't start until context is ready (a real dependency)
    return await output_guardrail(out)        # ← can't start until the model answers (a real dependency)

Two things to take away:

Same work, same cost — different schedule. Parallelizing doesn't make fewer calls or cheaper ones; it just stops steps from waiting on work they don't depend on. Pure latency win, no downside.
You can't parallelize past a real dependency. The model needs the retrieved context; the output guardrail needs the model's answer. That model → guardrail tail is sequential by nature — it's on the critical path, and shortening it needs a different lever (a faster/smaller model, streaming, L207–L212).

The Pipeline Lab below makes this physical: flip from sequential to orchestrated and watch the Gantt collapse the independent steps onto each other, the wall-clock drop ~38%, and the critical path light up. Internalize the rule: wall-clock = the longest dependency chain, not the sum of the steps.

Error Handling & Durability in the Pipeline

A pipeline that only handles the happy path is a demo. In production, steps fail — a provider 503s, a retrieval times out, a guardrail rejects an output — and the orchestrator decides what happens next. The core tools (several you've already met):

Retries with exponential backoff — a transient failure (429/5xx/timeout) is retried with growing delays before the step gives up.
Fallbacks — if a step keeps failing, take an alternate path: a backup provider (the gateway's fallback chain, L219), a cheaper model, a default response, or a human.
Circuit breakers — stop hammering a dead dependency: trip the breaker after repeated failures and fast-fail (or fall back) for a cooldown.
Conditional branching on failure — a downstream step reacts to an upstream error: skip the optional enrichment, run a compensating action (undo a partial write), or degrade gracefully.

For long-running or agentic pipelines (minutes to hours, dozens of steps), one failure shouldn't restart everything. That's durable execution: the orchestrator checkpoints state after each step and can resume from the last good point after a crash or restart. This is why graph/stateful orchestrators (LangGraph) and durable-workflow engines (Temporal-style) matter for serious agentic systems — they make a multi-step pipeline restartable instead of fragile.

The mindset: assume every step can fail, and decide the policy before it does. A pipeline's reliability isn't the average step working — it's what happens on the bad step. (And you only see which step failed if you have tracing — observability, L223.)

Framework or Plain Code?

The loudest question in this space: do you need an orchestration framework? The 2026 landscape, briefly:

Tool	What it's for
LangChain	General-purpose orchestration — connect LLMs to tools, APIs, DBs. Ubiquitous; some abstraction overhead.
LangGraph	Stateful, graph-based execution — cycles, persistence, human-in-the-loop. The go-to for complex / agentic flows.
LlamaIndex	Retrieval-centric — ingestion, indexing, query engines (great for the RAG part).
Haystack	Typed, visualizable DAG pipelines — every component has typed in/out; debug node by node.
DSPy	Programmatic prompt compilation — optimize prompts automatically instead of hand-tuning.

They're increasingly interoperable (LlamaIndex retrieval + LangGraph orchestration + LangSmith eval is a common production stack). But here's the advice that matters most, straight from Chip Huyen:

"While it's tempting to jump straight to an orchestration tool when starting a project, start building your application without one first."

Why? Because a framework you adopt before you understand your pipeline's shape costs more than it saves: hidden API calls, hidden latency, a leaky abstraction you fight, and a dependency that lags new model features. Plain code (a few async functions + asyncio.gather) is often all a 3–6 step pipeline needs — and it's fully transparent and debuggable.

Adopt a framework when it clearly earns its keep — judge it on Huyen's criteria:

Integration & extensibility — does it support your current and likely-future components, easily?
Complex-pipeline support — branching, parallelism, loops, error handling, state/persistence (this is where LangGraph shines and plain code gets painful).
Ease, performance & scalability — intuitive, well-documented, and no hidden latency or API calls.

The honest rule of thumb: plain code until the graph gets genuinely stateful or branchy; then reach for a graph orchestrator (LangGraph) — not before.

See It — The Pipeline Lab

Take one 8-component request pipeline and schedule it two ways. Flip between sequential and orchestrated (parallel) and watch the wall-clock — and the critical path:

The same 8-component request pipeline, scheduled two ways. **Sequential** runs every step one after another, so the wall-clock is the **sum** of all of them — even though the guardrail, router, and three retrievals don't depend on each other. Flip to **Orchestrated (parallel)** and watch the Gantt glide: the independent steps fan out and run **at once**, so the fan-out costs only its **slowest** branch. Wall-clock drops to the **critical path** (outlined) — the longest chain you *can't* parallelize: slowest retrieval → build prompt → **model** → output guardrail. Same work and same cost; ~**38% less latency** from a smarter schedule. The lesson: you parallelize the independent fan-out, but never past a real dependency. (Illustrative latencies.)

The Gantt tells the whole story. In sequential mode the bars sit end-to-end and the wall-clock is the sum (~2.45s) — with the guardrail, router, and retrievals idling in line. In orchestrated mode those independent bars slide to time-zero and overlap, the fan-out collapses to its slowest branch, and the wall-clock drops to the critical path (~1.53s, −38%) — the outlined chain slowest-retrieval → prompt → model → output guardrail that no schedule can shorten. Same work, smarter order.

🧪 Try It Yourself

Reason through these, then confirm with the Pipeline Lab:

Predict: in the lab, before flipping to parallel, which steps do you expect to "slide left" to the start, and which one becomes the slowest branch of the fan-out?
After parallelizing, the wall-clock is ~1.53s, not the ~0.8s of the model alone. Why can't orchestration make it any faster than that?
Parallelizing cut latency 38%. How much did it cut cost? Why?
You're starting a brand-new 4-step pipeline. A teammate wants to build it on a heavyweight orchestration framework "to be future-proof." What does this lesson say, and why?
Your agentic pipeline runs 20 steps over several minutes; step 14 crashes and the whole thing restarts from step 1. What capability was missing, and what kind of orchestrator provides it?

→ (1) The four independent steps — input guardrail, router, and the vector/web/SQL retrievals — slide to time-zero and run at once; web retrieval (450ms) is the slowest, so it sets the fan-out's cost. (2) Because of the critical path: model needs the context (so it can't start until the slowest retrieval + prompt finish), and the output guardrail needs the model's answer. That retrieval → prompt → model → guardrail chain is sequential by dependency — you can't parallelize past it. (3) Zero — it's the same calls, just scheduled differently. Parallelization is a pure latency win; it doesn't reduce the work, so cost is unchanged. (4) "Start building your application without one first." A 4-step pipeline is a few async functions; a heavy framework adds hidden latency, abstraction, and a dependency you'll fight before you even know your pipeline's shape. (5) Durable execution (checkpoint + resume) — provided by a stateful / durable-workflow orchestrator (LangGraph, Temporal-style), so a crash resumes from the last good step instead of restarting.

Mental-Model Corrections

"A pipeline is a straight line." It's a graph (DAG) — sequential, parallel, branching, and looping parts. The shape sets the latency, failure modes, and complexity.
"Parallelizing makes it cheaper." No — it's the same calls, rescheduled. Pure latency win (wall-clock = sum → critical path); cost is unchanged.
"Just parallelize everything to go fast." You can't parallelize past a real dependency — the model needs the context, the output guardrail needs the answer. Wall-clock floors at the critical path.
"Orchestration = LangChain." Orchestration is the job (define + chain components); LangChain is one tool for it. You can (and often should) do it in plain code first.
"Start on a framework to be future-proof." Backwards — "build your application without one first." Adopt a framework when it earns its keep (stateful/branchy graphs, persistence), judged on integration, pipeline support, and no hidden latency.
"Handle errors later." A pipeline's reliability is its failure handling — retries, fallbacks, circuit breakers, durable resume. Decide the policy before the step fails.
"The 5 workflow patterns are this lesson." Those (chaining/routing/parallel/orchestrator/evaluator) are the Agents material. This is the orchestration layer of the app architecture.

Key Takeaways

Orchestration is the glue that turns components into one end-to-end pipeline — Huyen's two stages: define the components, then chain them (sequence + pass data). It's where the architecture diagram becomes a program.
The pipeline is a graph (DAG), not a line — sequential, parallel, branching, and looping parts. The shape determines latency, failure modes, and complexity.
Parallelize for latency — the biggest cheap win. Run independent steps (guardrail, router, retrievals) concurrently → wall-clock drops from the sum to the critical path (~30–40% faster). Same work, same cost — and you can't parallelize past a real dependency.
Build in error handling & durability — retries w/ backoff, fallbacks (L219), circuit breakers, conditional branching, and durable execution (checkpoint/resume) for long agentic pipelines.
Framework or plain code: start in plain code (async + asyncio.gather); adopt LangGraph (stateful graph), LangChain, LlamaIndex (retrieval), Haystack (typed DAG), or DSPy (prompt compile) only when it earns its keep — and never one that hides latency or API calls.
Next — L222: Putting Guardrails in the Loop. We've wired the pipeline; now we make it safe — the input and output guardrails that protect it, placed exactly where they belong in the flow.