The Reference Architecture (Full Diagram)

You've spent two sections making the pieces of an LLM app fast (Inference & Latency, L207–L212) and cheap (Cost Optimization, L213–L217). This new section — AI Application Architecture — is where we assemble those pieces into a coherent system: the layers that turn a raw model call into a production application.

This lesson is the map. We'll lay out the full reference architecture — every layer, what it does, and how a request flows through all of them — and then the rest of the section zooms into each layer one at a time:

• The model gateway & router — the front door (L219)
• Adding caches — prompt + semantic (L220)
• Orchestration & pipelines — the glue (L221)
• Guardrails in the loop — the safety layer (L222)
• The monitoring & observability layer — the eyes (L223)

The single most important idea in this lesson (and it comes straight from Chip Huyen's Building a Generative AI Platform, the canonical reference here): you do not build all of this on day one. You start with the simplest thing that works — a bare model call — and add each layer only when you hit the specific problem it solves. "A component can be skipped if your system works well without it." Architecture is a response to pain, not a checklist.

So this lesson does two things: it gives you the whole picture (so you know what's possible and how the parts fit), and it teaches you the order to add things (so you don't over-engineer). Think of it as the blueprint you'll spend the next five lessons building, brick by brick.

Scope: this is the logical application architecture. The GPU-serving / deployment / autoscaling view (how the model box itself runs on hardware) is a later section (Deploying & Scaling) and was previewed in L209; RAG internals live in the RAG container; agent internals in the Agents material. Here we map how the layers fit together and defer each layer's depth to its own lesson.

Before the big diagram, internalize the starting point, because it's where every good LLM app begins:

user query  →  model  →  response

That's it. One call to a foundation model. No retrieval, no router, no cache, no guardrails. Most prototypes should look exactly like this, and a surprising number of shipped products never need much more.

Why start here instead of building the full architecture up front?

• You don't yet know where the pain is. Maybe the model hallucinates → you'll need context. Maybe it's too slow → cache. Maybe it's too expensive → routing. Maybe it leaks data → guardrails. Each layer is the cure for a specific symptom — and you can't prescribe a cure before you have the symptom.
• Every layer has a cost. More latency, more failure modes, more code, more money, more to monitor. A layer that doesn't solve a problem you actually have is pure liability.
• You can't optimize what you can't measure. The very first thing to add isn't a feature layer at all — it's observability (logs/traces), so the pain shows up as data instead of a hunch.

The rest of this lesson shows the destination — a fully-built architecture. Keep the journey in mind: you'll grow into this diagram one layer at a time. Treating it as a day-one checklist is the most common way teams over-engineer an app that a single model call would have served.

Here's the destination: a production LLM application is a model wrapped in layers. A request flows inward through the layers to the model and back outward through them to the user, while two layers — orchestration and observability — wrap the whole thing. Here's that request lifecycle, end to end:

Read top-to-bottom, that's the spine. Now the same eight layers as a map of responsibilities — and where each is covered:

The next five sections take the rows marked with a lesson number. The explorer below lets you click into all eight right now — but first, a quick tour of how they group.

The first thing a request hits isn't the model — it's the gateway. The gateway is a thin proxy that sits between your application and every model provider, and it exists so that cross-cutting concerns never leak into your app code: a single unified API, authentication, rate-limiting, cost caps, request logging, and fallbacks (retry a different provider when one is down). In 2026 it's considered critical infrastructure, not a nice-to-have — it's the one control point for all your AI traffic.

Riding alongside it is the router (the lever from L216): a small, fast classifier that reads the query, predicts what it needs, and sends it to the cheapest capable model or path — an FAQ lookup, a small finetuned model, a frontier model, or straight to a human. Together they form the app's entry layer: every request is authenticated, metered, and routed before any expensive work happens.

Gateway and router overlap with other layers on purpose — the gateway is also a natural home for caching and even some guardrails. That fluidity is normal; the architecture is modular, not rigid. L219 builds this layer in depth.

Inside the entry layer sits the core: the work of turning a query into a good answer, made accurate by context and fast/cheap by caching.

Context construction (RAG). A raw model only knows its frozen training data. The context layer retrieves the right external information — your docs, live data, private records — and tools, and adds them to the prompt so the model answers from facts, not guesses. As Huyen puts it, relevant context "can help the model generate more detailed responses while reducing hallucinations." (The full machinery — term/embedding/hybrid retrieval, rerankers, text-to-SQL, agentic retrieval — is the RAG container.)

Cache — "the most underrated component." Wrapped around (and inside) the core are three caches that stack, exactly the ones you metered in the Cost section:

• Prompt cache — reuse the repeated prefix (system prompt, long doc). If a 1,000-token system prompt rides on 1M calls/day, that's ~1B repeated input tokens/day you stop paying for. (L214)
• Exact cache — store the output of identical past requests (a summary, a SQL result).
• Semantic cache — reuse the answer for similar queries — powerful but "more dubious" (false positives), so evaluate before trusting it. (L215)

The core is where accuracy (context) and efficiency (cache) are won. L220 assembles the cache layer into the architecture; the RAG container is the context deep-dive.

Wrapping the model on both sides are guardrails — the layer that makes an app safe to put in front of real users and real systems.

• Input guardrails run before the model and protect against two things: leaking private data to an external API (the canonical cautionary tale: Samsung employees pasted proprietary code into a public chatbot and leaked company secrets) — mitigated by detecting and masking PII (reversibly) or blocking the query — and prompt injection / jailbreaks that try to make the model misbehave or run dangerous actions.
• Output guardrails run after the model and do two jobs: measure quality (empty, malformatted, toxic, hallucinated, brand-risk, or just bad responses) and specify the failure policy (retry — often in parallel to hide latency — or fall back to a human).

There's a real trade-off: guardrails add latency, and they break streaming (you can't fully vet tokens you've already shown the user). But "most teams find that the increased risks are more costly than the added latency."

Guardrails matter most when the system can take write actions (next layer) — an unguarded model with database access is a data-corruption incident waiting to happen. L222 is the deep-dive.

So far the request flows straight through. The agent loop adds the arrow that makes a system agentic: the model's output is fed back as the next input, enabling loops, planning, and recursive decomposition — "plan a weekend in Paris" → generate activities → iterate on each (hours, tickets, restaurants) → assemble.

The bigger leap is write actions: letting the model change the world — send an email, update a database, place an order — not just read and answer. That makes a system "vastly more capable" and vastly more dangerous: a prompt injection can now trigger a harmful write, a missing guard can corrupt production data, and database access can leak private info. The rule of thumb: "you shouldn't allow an unreliable AI to initiate bank transfers" — write actions demand the hardest guardrails and human approval gates.

The full agent toolkit (ReAct, planning, reflection, tool use, multi-agent) is the Agents material. Here, just place it on the map: it's the loop-back arrow plus the ability to act, and it raises the stakes on every other layer — especially guardrails and observability.

Two layers don't sit at one point in the flow — they wrap the whole thing.

Orchestration (→ L221) — the glue. Once you have more than query → model → response, something has to chain the components into an end-to-end pipeline: process the query → retrieve context → build the prompt → call the model → evaluate → return-or-escalate. Orchestration defines what components exist and the sequence (with branching, parallelism, and error handling) that connects them. Good orchestration parallelizes independent steps (run the router and the PII scrub at once) to claw back latency. Huyen's warning, though: "while it's tempting to jump straight to an orchestration tool when starting a project, start building your application without one first" — and avoid frameworks that hide API calls or inject latency.

Observability (→ L223) — the eyes. You can't operate what you can't see. Observability is three pillars: metrics (quality, latency — TTFT/TPOT from L207, cost), logs ("log everything"), and traces (a request's full path through every component, with per-step cost and latency, so a failure points you to the exact step that broke). "Integrate it from the beginning, not as an afterthought." A modern production trace spans 30–300 steps with cost attributed per step.

These two are "crucial for projects of all sizes," and their importance grows with complexity. They're drawn as bands wrapping the architecture because every other layer reports into observability and is sequenced by orchestration.

Here's the whole architecture as a clickable map. Open each of the eight layers to see what it does, when to add it, and which lesson goes deep — then read the footer:

Notice how each layer answers a different question: which model? (gateway/router), what does it know? (context), have we done this before? (cache), is this safe? (guardrails), are we done? (agent loop), how do the pieces connect? (orchestration), what's actually happening? (observability). A production app needs good answers to all seven — but it earns each layer by hitting the problem first.

The diagram is the destination, not the starting line. The discipline that separates good AI engineers from over-engineers is the order of assembly — add each layer only when a real symptom demands it:

Two rules make this concrete:

• Observability is the exception — add it first, not last. It's what turns vague pain ("feels slow") into a diagnosis ("the rerank step is the p95 bottleneck") so you add the right next layer.
• Evaluate at every step. Each layer can hurt (a bad router misroutes, a loose semantic cache serves wrong answers, a guardrail adds latency). Adding a layer without an eval to confirm it helped is how architectures rot.

The mature version of this architecture is genuinely complex — dozens of components, 30–300-span traces. But nobody designs it on a whiteboard and builds it all. It grows, one symptom-driven layer at a time, each one measured. That's the real lesson of the reference architecture: it's a map of where you might go, not a checklist of where you must start.

Reason through these, then confirm with the explorer:

1. Predict: you're prototyping a doc-Q&A bot. In what order would you add the layers, and which one would you add first of all?
2. Your bot confidently cites a policy that doesn't exist. Which layer fixes the cause, and which different layer catches it before the user sees it?
3. A request takes a slow, weird path and you have no idea which step is to blame. Which layer should have been there from the start?
4. You're tempted to start the project on a big orchestration framework "to be safe." What does this lesson (and Chip Huyen) say, and why?
5. Your agent gains the ability to send emails to customers. Which two layers immediately become non-negotiable?

→ (1) Start with just the Model API, and add observability first (logs/traces) so you can see what's happening. Then add layers by symptom — likely Context/RAG (it's a doc bot) next, then cache, guardrails, etc. (2) The cause is missing knowledge → add Context/RAG (ground it in real docs); the catch is an output guardrail (hallucination check) that blocks/flags it before the user sees it. (3) Observability — specifically tracing, which records the request's full path and the time/cost of each step so you can pinpoint the broken one. (4) Don't — "start building your application without one first." An orchestration framework you don't need yet adds hidden latency, hidden API calls, and complexity before you even know your pipeline's shape. (5) Guardrails (a write action + prompt injection can cause real-world harm) and observability (you must trace and audit every action it takes). Write actions raise the stakes on both.

• "Build the full architecture up front to be safe." Backwards. Start with one model call; add each layer only when a real symptom demands it. "A component can be skipped if your system works well without it."
• "The architecture is a fixed, linear pipeline." It's modular and fluid — layers overlap (the gateway also caches and guards), the agent loop bends the line back on itself, and orchestration/observability wrap everything rather than sitting at one point.
• "Observability is an ops concern to add at the end." Add it first. It's what turns pain into a diagnosis so you add the right next layer. "Integrate from the beginning."
• "The gateway and the router are the same thing." Related, not identical. The gateway is the unified, secure proxy (auth, limits, fallback); the router chooses which model/path per query. They sit together in the entry layer (L219).
• "More layers = better app." Every layer adds latency, failure modes, cost, and code. A layer that doesn't solve a problem you have is pure liability.
• "Caching and routing are cost tricks, not architecture." They're first-class layers of the production architecture (the most underrated one, even) — the Cost section was you building two of these layers.
• "Guardrails are optional polish." They're the difference between a demo and something you can put in front of users — and they're mandatory the moment the system can take write actions.

• A production LLM app is a model wrapped in layers. A request flows in through the gateway, input guardrails, router, context/RAG + cache, to the model, then out through output guardrails (with an agent loop if multi-step) — all chained by orchestration and watched by observability.
• The eight layers, and where they're built: Gateway & Router (L219), Context/RAG (RAG container), Cache (L220, + L214/L215), Guardrails (L222), Agent loop (Agents material), Orchestration (L221), Observability (L223) — wrapping the Model (Inference section).
• Start simple; grow by symptom. Begin with query → model → response. Add context when it hallucinates, cache when it's slow/costly, gateway/router when models multiply, guardrails when it's user-facing, agents when one call can't finish, orchestration when the glue gets tangled.
• Two layers are special: add observability from day one (it diagnoses which layer to add next), and don't reach for an orchestration framework first ("build your application without one first").
• Evaluate at every step — each layer can hurt (misroutes, false-positive caches, added latency), so confirm each one helps. The architecture grows, measured, one layer at a time.
• Next — L219: The Model Gateway & Router. We start building, from the front door inward: the one secure, unified entry point for all your model traffic, and the classifier that routes each request to the right model.