Deploy & Observe

Introduction

This is the finale of Project 1. You've built a real RAG assistant — it retrieves (L2), grounds and cites with abstention (L3), and you can prove its quality (L4). One thing remains: ship it. A system on your laptop isn't a portfolio piece; a system running at a public URL that you can watch in production is. This lesson does both halves of shipping:

Deploy — turn the FastAPI app into a real service on Modal: a public HTTPS /ask endpoint, secrets from the environment, and the production knobs (warm pools, autoscaling, concurrency) that decide your latency and bill.
Observe — wire Arize Phoenix so every request emits a trace: a span per step (retrieve → rerank → generate) with its latency, tokens, cost, and the exact chunks retrieved — so when something is slow, expensive, or wrong, you can see why.

Why both, and why this is the lesson that makes it real. A deployed system you can't see into is a black box — when a user gets a bad answer, you have "200 OK, 1.9s" and nothing else. And observability without deployment is academic. Together they're the difference between "I built a RAG demo" and "I run a RAG service in production and here's its trace, its p95, and its cost per query." That sentence is what gets you hired.

We use the course's own stack: FastAPI on Modal for serving (serverless, scales to zero, deploys in one command), and Phoenix + OpenInference + OpenTelemetry for observability (it auto-instruments the Claude call and lets you add spans for retrieval). By the end, you'll have a live URL and a trace dashboard — and Project 1 will be a thing you can demo.

Hero infographic titled 'Deploy & Observe' for the fifth and final lesson of the Production RAG Assistant capstone, on a white background. The deck says: ship the assistant as a service on Modal, then watch every request with Phoenix — the last step of Project 1. The figure has two halves joined in the middle. The left half, DEPLOY, shows your FastAPI app wrapped by a Modal asgi_app decorator and deployed to a serverless container that exposes a public HTTPS /ask endpoint; callouts read warm pool (min_containers ≥ 1) to kill cold starts, autoscale (max_containers), concurrency, and secrets from the environment. A user hits the /ask URL. The right half, OBSERVE, shows that same request emitting a distributed trace into Phoenix: a waterfall of spans — chain /ask, then retrieve (RETRIEVER), rerank (RERANKER), and generate (an LLM span auto-instrumented by OpenInference for the Claude call) — each bar's width is its latency, with token count, cost, and a quality flag on the span. A LATENCY BUDGET strip shows where the milliseconds go: retrieve, rerank, and generate, with generation dominating, plus a time-to-first-token marker and a note that streaming masks generation latency. A small panel notes p50, p95, p99 tail latency and dollars per query. A closing note reads: online evals on sampled live traces catch a 200 OK that is still wrong. Three summary cards along the bottom read: 'Deploy — FastAPI on Modal, warm pool kills cold starts'; 'Observe — a Phoenix trace per request, span-level latency + cost + quality'; 'Close the loop — online evals on live traffic'. A family strip lists the five build lessons — Brief & Architecture, Ingestion & Hybrid Retrieval, Generation Citations & Guardrails, Building the Eval Harness, Deploy & Observe — with the fifth highlighted. Slate, sky, cyan, violet, amber, emerald accents; generous legible text.

The Two Halves of Shipping

"It works on my machine" is where most projects stop and most learning is wasted. Production is a different discipline with two requirements:

1. Run it as a service. Someone (a frontend, another service, an interviewer with your URL) must be able to POST a question and get an answer over HTTP, reliably, without you babysitting a uvicorn in a terminal. That means a real host, secrets handled safely, and behavior under concurrent load and cost that you've thought about.

2. See what it does. The moment real users hit it, you need to answer: which step was slow? what did it cost? what did it actually retrieve? was the answer even right? A traditional log line (200 OK · 1.9s) answers none of these — and in LLM systems, a 200 OK can still be a hallucination. You need tracing, not just metrics.

The beautiful part: on the course stack, both halves are a few lines. Modal turns a Python file into a deployed, autoscaling HTTPS service with one command; Phoenix turns your pipeline into a fully traced system with one register() call. The rest of this lesson is making each one production-grade — and understanding the trade-offs (cold starts, latency budgets, what to put on a span) that separate a demo deploy from a real one.

Deploying on Modal

Modal runs your Python in the cloud with no Dockerfiles, no YAML, no Kubernetes — you describe the image and the function in Python, and modal deploy gives you a live HTTPS URL. To serve your FastAPI app, wrap it with @modal.asgi_app(): Modal routes web traffic to the ASGI app you return. Define the image (deps from your pyproject profiles), mount your rag package, and load secrets:

# modal_app.py   ->  `modal deploy modal_app.py`
import modal

# Build the container image from your dependency profiles (L1) + mount the rag package.
image = (
    modal.Image.debian_slim(python_version="3.11")
    .pip_install_from_pyproject("pyproject.toml", optional_dependencies=["retrieve", "obs"])
    .add_local_python_source("rag")
)
app = modal.App("rag-assistant", image=image)


@app.function(
    secrets=[modal.Secret.from_name("rag-secrets")],   # ANTHROPIC_API_KEY, VOYAGE_API_KEY, QDRANT_URL...
)
@modal.asgi_app()
def fastapi_app():
    from rag.observability import setup_tracing
    setup_tracing()                  # wire Phoenix BEFORE the app serves (next sections)
    from rag.api import app as web_app
    return web_app                   # the FastAPI app from L1/L3 (/health, /ask)

Secrets, the right way. modal.Secret.from_name("rag-secrets") injects your keys as environment variables at runtime — your Settings (L1) reads them from env, so nothing is hard-coded and nothing is committed. Create the secret once with modal secret create rag-secrets ANTHROPIC_API_KEY=... VOYAGE_API_KEY=....

Then: modal serve modal_app.py for a hot-reloading dev URL, and modal deploy modal_app.py for the real, persistent HTTPS endpoint. Your assistant is now on the internet.

Production Modal — Cold Starts, Warm Pools, Concurrency

The default deploy scales to zero — cheap, but the first request after idle pays a cold start (the container boots and imports your code, often 1–2 seconds). For an interactive assistant, that cold start lands in your tail latency (p95/p99) and feels terrible. Modal gives you three knobs to trade cost for latency:

@app.function(
    secrets=[modal.Secret.from_name("rag-secrets")],
    min_containers=1,        # WARM POOL: keep >=1 container alive -> no cold start on the hot path
    max_containers=10,       # autoscale ceiling (cost guardrail)
    scaledown_window=300,    # keep idle containers 5 min before scaling down
)
@modal.concurrent(max_inputs=20)   # one container serves up to 20 concurrent requests
@modal.asgi_app()
def fastapi_app():
    ...

min_containers=1 is the warm pool — the single highest-impact production setting for an LLM API. It keeps a container alive so requests skip the cold start; you pay for one idle container in exchange for a clean tail latency. (In the console below, toggle the warm pool and watch p95 drop below the SLO.)
@modal.concurrent(max_inputs=20) lets one container handle many requests at once — which is perfect here because your endpoint is I/O-bound (it spends its time waiting on Qdrant, Voyage, and Claude), so a single container can serve many in-flight requests cheaply.
max_containers caps autoscaling so a traffic spike (or a bug) can't run up an unbounded bill.

(Modal 1.0 names: min_containers was keep_warm, max_containers was concurrency_limit, scaledown_window was container_idle_timeout.) These are the trade-offs you'll defend in an interview: warm pools and concurrency buy latency; caps buy cost safety.

The Latency Budget — Where the Milliseconds Go

Before you optimize, measure where the time goes — and report percentiles, not averages (an average hides the p95/p99 tail that real users feel). A typical RAG /ask breaks down like this:

Retrieve (hybrid + RRF): tens of ms — cheap. (Retrieval is ~40% of time-to-first-token in many production setups once you add multi-stage search.)
Rerank (cross-encoder): ~100–300 ms — a real, visible chunk; the price of precision.
Generate (Claude): the dominant cost — split into TTFT (time to first token, a few hundred ms) and decode (TPOT × output tokens, often seconds).
Cold start (if no warm pool): 1–2 s bolted onto the front — pure tail-latency poison.

Two moves tame this. First, stream. Generation dominates total latency, but with streaming the user sees the first token in well under a second and reads as the rest arrives — so perceived latency (TTFT) is a fraction of the total. Streaming is the single biggest perceived-speed win, and it's free (you already saw messages.stream in L3). Second, kill the cold start with the warm pool. Track TTFT and end-to-end p95 as separate SLOs — "first token < 1 s, full answer p95 < 5 s" — because they're improved by different levers. The console below makes this budget tangible: watch the trace bars, the TTFT marker, and the p95-vs-SLO verdict move as you flip streaming and the warm pool.

Observability — Why a Trace, Not a Log

Now the other half. Traditional monitoring gives you a black box: POST /ask → 200 OK · 1.9s. That tells you the request finished — not why it was slow, what it cost, which chunks it retrieved, or — critically — whether the answer was right. In an LLM system, "200 OK" and a hallucination look identical to your web server.

The fix is a trace: a tree of spans, one per pipeline step, each carrying its latency, tokens/cost, inputs/outputs, and quality signals. A trace turns the black box into a glass box — you can point at the exact span that was slow (the reranker had a retry), the span that drove cost (a big-context generate call), and the span that retrieved the wrong chunk. The slow step and the expensive step are often not the same, and you can only tell them apart with span-level data.

This is also how you close the loop with L4: a flagged production span — a low-faithfulness answer on real traffic — becomes a new example in your golden set. Observability and evaluation are the same discipline, one offline (eval) and one online (tracing). Both ask: what is this system actually doing?

Wiring Phoenix — OpenInference + OpenTelemetry

Phoenix is the open-source LLM observability platform on the course stack; it's built on OpenTelemetry (the industry transport for traces) and OpenInference (LLM-specific conventions, so Phoenix knows which spans are LLM calls vs. retrievals). Setup is two steps: register() a tracer pointed at Phoenix, then turn on the Anthropic auto-instrumentor — which traces every Claude call for free (model, tokens, latency, cost):

# src/rag/observability.py
from openinference.instrumentation.anthropic import AnthropicInstrumentor
from phoenix.otel import register

from rag.config import settings


def setup_tracing():
    tracer_provider = register(
        project_name="rag-assistant",
        endpoint=settings().phoenix_endpoint,   # Phoenix Cloud OTLP, or http://localhost:6006/v1/traces
        auto_instrument=True,
    )
    # auto-instrument the Claude SDK -> every messages.create()/.stream() becomes an LLM span
    AnthropicInstrumentor().instrument(tracer_provider=tracer_provider)
    return tracer_provider.get_tracer("rag")


tracer = setup_tracing()

The Claude call is now traced automatically. But Phoenix can't see your retrieval — that's your code, not the SDK's. Add a manual span and mark it as a RETRIEVER using OpenInference conventions, so Phoenix renders it as a retrieval step (with the documents it returned):

# src/rag/retrieve/__init__.py  (instrument the retrieve() from L2)
from openinference.semconv.trace import OpenInferenceSpanKindValues, SpanAttributes

from rag.observability import tracer
from rag.retrieve.hybrid import hybrid_search
from rag.retrieve.rerank import rerank


def retrieve(query: str) -> list[dict]:
    with tracer.start_as_current_span("retrieve") as span:
        span.set_attribute(SpanAttributes.OPENINFERENCE_SPAN_KIND,
                           OpenInferenceSpanKindValues.RETRIEVER.value)
        span.set_attribute(SpanAttributes.INPUT_VALUE, query)
        hits = rerank(query, hybrid_search(query))      # L2 pipeline
        for i, h in enumerate(hits):                    # record what was retrieved + scores
            span.set_attribute(f"retrieval.documents.{i}.document.id", h["id"])
            span.set_attribute(f"retrieval.documents.{i}.document.score", h["rerank_score"])
        return hits

Now a single /ask produces a full trace — chain → retrieve (RETRIEVER) → generate (LLM) — visible in Phoenix as a waterfall. (register() can also auto-instrument FastAPI so the HTTP request is the root chain span.)

What to Put on a Span

A trace is only as useful as what you record on it. The auto-instrumentor captures the LLM basics (model, tokens, cost, latency); you add the RAG-specific signals that let you debug quality, not just speed. For a production RAG assistant, capture:

On the retrieve span: the query, the retrieved chunk ids + scores, and the top rerank score (so you can see exactly what context the answer was built from — the #1 debugging question).
On the generate span: input/output tokens and cost (auto), plus the citations and the abstained flag from L3 — so you can find answers that were ungrounded or that refused.
On the root /ask span: the final answer, total latency, and a place for an online evaluation score (next).

Cost attribution. Roll token counts up to the trace and tag them by model and (if multi-tenant) by user/feature — this is how "our bill went up" becomes "the new long-context prompt on the summarize feature doubled output tokens." A caution: captured inputs/outputs may contain PII — Phoenix lets you turn off content capture (keep metadata, drop the text) for sensitive deployments. Trace what you need to debug, not everything.

Closing the Loop — Online Evaluation

Offline eval (L4) scores a fixed golden set. But production sees questions you never imagined — and a 200 OK can still be a hallucination. The final piece is online evaluation: run your eval metrics on sampled live traces, continuously, so a silent quality regression on real traffic is caught, not discovered via an angry user.

The neat part: a production trace already contains everything a reference-free metric needs — the query, the retrieved contexts, and the response. So you can score live traffic for faithfulness and answer-relevancy (the two RAGAS metrics that need no ground truth) right on the spans Phoenix collected:

# src/rag/eval/online.py   ->  run on a sample of live traces (e.g. a Modal cron)
from ragas import SingleTurnSample
from ragas.metrics import Faithfulness          # reference-FREE: works on live traffic

from rag.eval.judge import judge_llm


async def score_live_trace(query: str, contexts: list[str], answer: str) -> float:
    sample = SingleTurnSample(
        user_input=query, retrieved_contexts=contexts, response=answer  # no reference needed
    )
    metric = Faithfulness(llm=judge_llm)
    score = await metric.single_turn_ascore(sample)
    # low score -> alert + add this real example to the golden set (closes the loop to L4)
    return score

Alert on the SLOs that matter — not just "is it up?" but p95 latency, cost/query, and online faithfulness. A faithfulness dip after a deploy is the LLM-native incident the web server can't see. And every low-scoring live example you find becomes a new golden-set entry — so the system gets better from its own production traffic. That's the loop closing: deploy → observe → evaluate → improve → deploy.

✅ Definition of Done (this step & the project)

This is the last step — finish it and Project 1 is shippable.

Deployed on Modal — modal deploy modal_app.py gives a public HTTPS /ask; secrets from env (nothing hard-coded).
Production config — a warm pool (min_containers≥1), a max_containers cap, and @modal.concurrent for the I/O-bound endpoint.
Streaming on /ask, and you can state your TTFT and p95 SLOs.
Phoenix wired — register() + AnthropicInstrumentor; a /ask produces a trace (chain → retrieve RETRIEVER → generate LLM).
Rich spans — retrieved chunk ids + scores, tokens, cost, citations, abstained on the right spans.
Online eval — faithfulness scored on sampled live traces, alerting on regressions.
README updated — the architecture, the stack decisions, the eval scores, the live URL, a screenshot of a Phoenix trace. This is your portfolio write-up.

If an interviewer can open your URL, ask a question, get a cited answer, and you can pull up the trace that produced it — you've built something the vast majority of "I know RAG" candidates cannot show. Project 1: done.

See It — The Deploy & Observe Console

This console fuses the two halves of this lesson. Set the deploy config and watch a Phoenix-style trace of a POST /ask recompute live.

Start on scale-to-zero. A ~1.9 s cold start dominates the trace and pushes p95 over the SLO (red). Toggle the warm pool and the cold-start span vanishes — p95 clears the SLO.
Toggle streaming. Watch perceived TTFT collapse (first token in ~1 s) while the total is unchanged — same work, far better UX. Turn it off and the user waits for the whole answer.
Switch Sonnet → Haiku. Latency and $/query both drop — the lever you'd reach for if the eval set from L4 says quality holds.
Read the waterfall. Generate is the widest bar and ~90% of cost; retrieve and rerank are small but real. The trace is how you'd see this in production.

The Deploy & Observe Console — the two halves of shipping this assistant, made playable. Set the deploy config — Streaming, Warm pool (min_containers ≥ 1 vs scale-to-zero cold starts), generation model (Sonnet/Haiku), Reranker — and watch a live Phoenix-style trace waterfall of a POST /ask recompute: a chain with retrieve (RETRIEVER), rerank (RERANKER), and generate (LLM) spans, each bar's width its latency, with a 'first token' marker. A readout shows perceived TTFT, total latency, a p95 estimate vs the SLO, and dollars per query. Start on scale-to-zero: a ~1.9s cold start dominates the trace and blows p95 past the SLO. Turn on the warm pool and the cold-start span vanishes — p95 clears the SLO. Toggle streaming and watch perceived time-to-first-token collapse while the total is unchanged (same work, far better UX). Switch Sonnet → Haiku and watch latency and cost drop. The lessons: generation dominates both latency and cost, cold starts live in the tail, and streaming buys perceived speed for free. Latencies are a deterministic teaching model, not live calls.

Notice three things. One: the cold start lives in the tail — the warm pool is the highest-impact latency setting. Two: streaming buys perceived speed for free — total latency is the same, but the user feels the TTFT. Three: generation dominates both latency and cost, so it's where model choice (Sonnet vs. Haiku) pays off — measured against your eval set.

🧪 Try It Yourself

Reason these out, then check against the console and the code.

1. Your /ask p50 is a snappy 1.2 s but p95 is 3.4 s and users complain it's "sometimes really slow." What's the most likely cause, and which one Modal setting fixes it?

2. Generation takes 3 s and you can't make Claude faster. How do you make the assistant feel fast anyway, and what metric does that improve (vs. leave unchanged)?

3. A user reports a wrong answer. Your web logs say 200 OK · 2.1s. Why is that useless, and what would a trace let you check that the log can't?

4. Why mark the retrieval span as a RETRIEVER (OpenInference) and record the chunk ids + scores on it, instead of just letting the auto-instrumentor trace the Claude call?

5. You run faithfulness on a sample of live traces and it's dropping this week, though latency and error rate are fine. What happened, why didn't ordinary monitoring catch it, and what do you do with the low-scoring examples?

Answers.

1. Cold starts. A scale-to-zero deployment boots a fresh container (1–2 s) for the first request after idle — those land in the tail (p95/p99) while warm requests stay fast (p50). Set min_containers=1 (a warm pool) so requests skip the cold start. (Averages would have hidden this — always watch percentiles.)

2. Stream the response. With streaming the user sees the first token in a few hundred ms and reads as the rest arrives, so perceived latency (TTFT) drops dramatically — even though total end-to-end latency is unchanged. It's the biggest perceived-speed win and essentially free.

3. 200 OK · 2.1s tells you it finished, not why it was slow, what it cost, what it retrieved, or whether the answer was right — a hallucination is also a 200 OK. A trace shows the span-level breakdown: which step was slow, the cost-driver span, and — most importantly — the exact chunks retrieved and the answer, so you can see whether retrieval missed or the model drifted.

4. Because the auto-instrumentor only sees the Claude SDK call — your retrieval is your own code and is invisible to it. Marking a manual span as RETRIEVER makes Phoenix render it as a retrieval step, and recording the chunk ids + scores answers the #1 debugging question — "what context did the answer come from?" — directly on the trace.

5. A silent quality regression — a change (a model version bump, a prompt edit, drifting data) made answers less grounded, but they're still well-formed and return 200 OK quickly, so latency and error-rate monitoring see nothing. Online faithfulness is the only signal that catches it. Take the low-scoring live examples and add them to your golden set (L4) — now your offline eval guards against that failure forever. The loop closes.

Mental-Model Corrections

"It runs locally, so it's done." → Production is a separate discipline: serve it (a real host, secrets, concurrency, cost) and observe it (traces). A localhost demo isn't a portfolio piece.
"Average latency looks fine." → Averages hide the tail. Watch p95/p99 — that's where cold starts and retries live, and what users actually feel.
"Generation is slow, nothing I can do." → Stream it. You can't cut total latency, but streaming slashes perceived latency (TTFT) for free.
"Cold starts are unavoidable on serverless." → A warm pool (min_containers≥1) removes them from the hot path — one idle container's cost buys a clean tail latency.
"Logging is observability." → 200 OK · 1.9s tells you nothing about why or whether it was right. You need span-level traces with latency, cost, retrieved chunks, and quality.
"A 200 OK means it worked." → In LLM systems a 200 OK can be a hallucination. Health checks and error rates don't see quality — online evals do.
"Observability and evaluation are different things." → They're the same question (what is this system doing?) — offline on a golden set, online on live traces. Low-scoring production traces feed the golden set; the loop closes.
"Trace everything." → Capture what you need to debug (chunk ids, scores, tokens, cost, quality), and mind PII — you can keep metadata and drop sensitive content.

Key Takeaways — and Project 1 Complete

Shipping is two halves: deploy + observe. A service the world can hit, and the ability to see what it does in production.
Deploy on Modal: wrap FastAPI with @modal.asgi_app(), secrets from modal.Secret, then modal deploy. Make it production-grade with a warm pool (min_containers≥1), max_containers, and @modal.concurrent for the I/O-bound endpoint.
Mind the latency budget: report p95/p99, not averages; stream to crush perceived TTFT; warm pools to kill cold-start tails. Generation dominates total latency and cost.
Observe with Phoenix: register() + AnthropicInstrumentor traces Claude for free; add a manual RETRIEVER span. A trace (latency + cost + retrieved chunks + quality) turns a black box into a glass box — a 200 OK can still be a hallucination.
Close the loop with online eval: score faithfulness on sampled live traces (reference-free), alert on regressions, and feed low-scoring real examples back into the golden set (L4).

🎉 Project 1 is complete. You've built — and can defend — a production RAG assistant over your own documents: hybrid retrieval + reranking (L2), grounded cited generation with abstention (L3), a RAGAS evaluation harness (L4), and a deployed, observed service (L5). That's the most common real AI system in industry, end to end, with a live URL and a trace dashboard. Put it on your résumé.

Next — Project 2: a Multi-Tool Agent with MCP. Where RAG retrieves and answers, an agent reasons and acts — calling tools, holding memory, and orchestrating multi-step work over the Model Context Protocol. You'll bring everything you just learned (grounding, evaluation, observability, deployment) to a system that does things, not just answers. On to Section 2.