Wiring Up Observability End-to-End

This is the capstone of Deploying & Scaling. You designed the API (L244), set rate limits (L245), gated changes with evals in CI (L246), versioned every artifact (L247), and built the scaling & reliability patterns (L248). Now you ship it — and the moment real traffic hits, you face the operator's question that none of those lessons answered: what is actually happening in there, right now?

A deployed LLM app is a distributed pipeline of non-deterministic, billed, failure-prone steps — gateway, retrieval, rerank, model call, tools, guardrails, sub-agents. When a request is slow, expensive, or wrong, ‘200 OK in 1.9s’ tells you nothing about which step did it. Worse: a model that confidently hallucinates returns a perfect 200 OK — your dashboards stay green while quality rots. You cannot operate, debug, or improve what you cannot see. This lesson is about wiring up that sight, end to end.

In this lesson:

• The four signals you actually operate on — metrics, traces, logs, and evals
• Instrumenting with OpenTelemetry and the GenAI semantic conventions (vendor-neutral)
• Why the trace is the spine — and how context propagation turns six services into one picture
• What to emit at each hop (tokens, cost, TTFT, version) and the PII problem in logs
• Online evals & drift — scoring live traffic so quality is a signal, not a vibe
• Alerting on SLOs and error-budget burn (L248), and closing the loop from a bad trace to a fix

Scope: L223 (The Monitoring & Observability Layer) already made the conceptual case — why LLM observability ≠ traditional monitoring, the pillars, and traces as the centerpiece. This lesson is the hands-on wiring and operation: how you instrument it, propagate it, alert on it, and run it in production. The deep comparison of specific platforms (LangSmith, Langfuse, Phoenix, Braintrust, Helicone) is its own dedicated lesson — here the tools are interchangeable behind the OpenTelemetry standard. This is the last lesson of the Deploying & Scaling section; next you start a new pillar, Multimodal AI Engineering (L250).

Traditional monitoring answers one question — is the service up and how slow is it? For an LLM app that's necessary but nowhere near sufficient, because the failure that hurts you most — a wrong but confident answer — is invisible to uptime and latency. So production LLM observability is built on four complementary signals, and you need all four:

Metrics tell you a number moved. Traces tell you which hop moved it. Logs tell you what that hop saw. Evals tell you whether the output was correct, grounded, and safe — the dimension classic monitoring has no opinion on. The rest of this lesson wires up each one and then operates them together; the interactive at the end lets you break each signal and watch the others compensate (or fail to).

Don't hand-roll a logging format you'll regret. The industry has standardized on OpenTelemetry (OTel) — the vendor-neutral standard for traces, metrics, and logs — and since 2024 the GenAI Semantic Conventions define a common vocabulary for LLM telemetry so every tool produces consistent data. Instrument once against the standard, and you can send it to any backend (Langfuse, Phoenix, Datadog, Grafana…) and switch later without re-instrumenting.

The conventions name an LLM call a span — typically named {operation} {model} (e.g. chat gpt-5.5) — carrying standard attributes: gen_ai.system (the provider), gen_ai.request.model, gen_ai.usage.input_tokens / output_tokens, and gen_ai.response.finish_reasons. (Note the 2025 spec deprecated the old gen_ai.prompt / gen_ai.completion attributes in favor of structured gen_ai.input.messages / output.messages.) You rarely write this by hand — auto-instrumentation libraries (OpenLLMetry, and the providers' own OTel integrations) wrap the SDK calls for you — but knowing the shape is what lets you add the custom spans for your retrieval, rerank, and tool hops:

Spans are only useful if they join up. The magic of distributed tracing is a single trace id that flows through every service in a request so all their spans assemble into one waterfall — parent and children, with exact timings. That's what turns ‘the request was slow’ into ‘the web retrieval span took 1.2s — there's your bottleneck.’

The mechanism is context propagation: the trace id (and parent span id) ride along with the request, in-process automatically and across network boundaries via standard W3C traceparent headers. When your gateway calls retrieval, retrieval calls the model, the model triggers a tool, and an agent hands off to a sub-agent — the context propagates through all of it, so the whole chain is one trace, not seven disconnected logs. (Real pipelines run 30–300 spans per request.)

The failure mode to burn into memory: an uninstrumented hop is a blind spot. If retrieval doesn't emit a span — or context propagation breaks at a boundary — that step becomes a black hole in the trace. You'll see that the request was slow but not where, and your mean-time-to-resolution goes from minutes to hours of guessing. You cannot debug what you did not instrument. In the interactive, turn a hop dark and watch the bottleneck vanish from the trace while the alert still fires.

A generic web trace records status and duration. An LLM trace must also capture the things that drive cost and quality, per span:

• Tokens & cost — input_tokens / output_tokens and the dollar cost per call. This is how you attribute spend to a feature, a user, or a runaway prompt — the model call is almost always the cost driver, and a single bloated context shows up here instantly.
• Latency, broken down — overall latency and the LLM-specific pieces: TTFT (time to first token — what the user feels, L225), inter-token latency, and retrieval latency for RAG. TTFT and generation time swing with output length, model load, batching, and KV-cache hits (L208/L209).
• The exact versions — gen_ai.request.model pinned to a snapshot, plus your prompt version and RAG-config version (L247). When quality moves, the first question is ‘what changed?’ — and the trace must carry the answer.
• Outcome flags — finish_reason (did it stop cleanly, hit max tokens, or call a tool?), cache hit/miss (L214/L220), guardrail verdicts (L231/L232), and retry/fallback events (L248).

A practical split keeps the bill sane: emit metrics for 100% of requests (tokens, cost, latency, error rate are cheap counters) but sample the detailed traces — typically capture full traces for 10–20% of normal traffic while tail-sampling keeps 100% of slow and errored requests, because those are the ones you'll actually want to open.

Logs are the third signal: the structured record of what the model actually saw and said — the rendered prompt, the retrieved chunks, tool inputs/outputs, the raw completion. They're indispensable for debugging ‘why did it answer that?’ — but they're also the most dangerous data you hold, because prompts and outputs are full of user PII, secrets, and proprietary content, and you're about to copy all of it into a third-party observability backend.

Treat logging as a governance problem, not just a feature:

• Redact before you store. Run PII detection/redaction (the same machinery as your input guardrails, L231 — Presidio, etc.) on prompts and outputs before they leave for the telemetry pipeline.
• Differential logging. Send full prompt/response bodies only to a restricted-access store; surface only metadata (token counts, latency, scores, hashes) to the general dashboards everyone sees.
• Hash for pattern analysis. A prompt hash lets you group identical prompts and spot caching opportunities or repeated abuse without storing the text.
• Sample and set retention. You don't need every body forever — sample, and expire on a schedule that matches your compliance posture (GDPR/CCPA, L236).

This connects straight to your audit log (L237): the audit trail is the tamper-evident, compliance-grade subset of your logs, while general observability logs are the high-volume, redacted, sampled operational view.

Here's the signal that makes LLM observability different — and the one most teams skip until it burns them. Metrics, traces, and logs can all be green while your answers quietly get worse. Quality is invisible to uptime, latency, and cost — a hallucination is a fast, cheap, 200 OK. The only way to see it is to score the output, on live traffic. That's online evaluation.

You already built evals for CI (L246) — a fixed test set, run pre-deploy, gating merges. Online evals are the same scorers pointed at production: on a sampled slice of real traffic (1–10%, scored asynchronously so you never add latency to the user's request), run LLM-as-judge plus cheap heuristics (format, grounding/faithfulness, length) plus your guardrail signals, and emit the score as a metric you can alert on. Offline evals catch the cases you thought of; online evals catch the drift, edge cases, and regressions a static set never imagined.

Two honest caveats:

• LLM-as-judge is a noisy estimator, not ground truth — it has biases and inconsistency. Calibrate it against human labels and trend it over time rather than trusting any single score (L233's judge cautions apply).
• Watch for drift explicitly. Inputs and outputs shift over weeks (new user behavior, a silent provider model update). Track distributional drift — PSI, KL or Jensen–Shannon divergence on token distributions, embedding centroids, or the eval-score series — and alert when it strays from a rolling baseline, before it becomes a crisis.

Signals are useless if nobody's told when they go bad — and noisy alerts are worse than none (everyone learns to ignore the pager). The discipline from L248 carries straight over: alert on your SLOs and the rate you're burning the error budget, not on raw thresholds that flap.

Two rules keep alerts trustworthy:

• Burn-rate, multi-window. Don't page because one request was slow. Page when you're burning the error budget fast — e.g. a metric is breaching SLO over both a short and a long window — which catches real outages while ignoring blips.
• Tier by severity. A safety/guardrail eval firing → page immediately. A quality drift (judge score >10% below the rolling 7-day baseline across enough sampled traces over 30 min) → a ticket, not a 3am page. Latency/error SLO burn → page on fast burn, ticket on slow.

And keep logging separate from evaluating: log everything immediately and run evals asynchronously so an eval backlog can never slow or break the serving path.

Observability isn't a wall of dashboards you glance at; it's the engine of improvement that ties this whole course together. A single flagged production trace should flow through your entire machine:

1. Online eval flags a bad trace (low judge score, failed grounding, a guardrail hit).
2. The trace becomes an eval case — added to your offline set (L246) so that exact failure is now regression-tested forever (the observability → evaluation loop from L223).
3. If it's serious, it opens an incident with the trace as evidence, alongside your tamper-evident audit log (L237).
4. You ship a fix — a prompt edit, a retrieval tweak, a model pin — as a new version (L247), gated by the now-richer eval set in CI (L246), and canary it (L246/L248) while online evals watch the rollout.

That loop — observe → evaluate → fix → version → re-deploy → observe — is what makes a production AI system get better over time instead of silently decaying. Observability is the sensor that closes it.

Stop reading, start operating. The Observability Console is a live LLM pipeline you both wire up and run under incidents. Click any hop to toggle its instrumentation, flip online evals on or off, set the trace sampling rate — then inject an incident and read the three linked views: the distributed trace, the signals vs SLO, and the alert + root-cause verdict.

Three experiments that teach the lesson:

1. Make a blind spot. Inject a latency spike, then click the retrieve hop to turn its span off. The alert still fires (p95 breached) — but the slow span vanishes into a ‘? blind’ gap. You know something's wrong, not where. You can't debug what you didn't instrument.
2. The silent regression. Inject a quality drop with online evals ON — the LLM span flags low quality and you root-cause it. Now turn evals OFF: latency, errors, and cost stay green, the response is a healthy-looking 200 OK, and no alert fires — the hallucination is invisible. Online evals are the only signal that catches it.
3. Sample sanely. Drag trace sampling down — note metrics stay at 100% and slow/error traces are still kept (tail sampling), so you cut cost without going blind on the traces that matter.

The takeaway you operate by: metrics say something's wrong, the trace says where, logs say what the model saw, and evals say whether it was any good.

Reason it through (then check in the console):

1. Your p95 alert fires but every span in the trace looks fast. What's the most likely cause — and what does it tell you about your instrumentation?
2. A user reports a wrong answer. Your metrics dashboard is all green. Which signal would have caught this, and why didn't the others?
3. You're paying a fortune to store traces. How do you cut the bill without losing the ability to debug incidents?
4. Why alert on error-budget burn rate instead of ‘p95 > 4s’ directly?

Answers:

1. A blind spot — the slow work is happening in a hop that isn't instrumented (or context propagation broke at a boundary), so its time doesn't appear in the trace. The fix is to instrument that hop; the lesson is that a green-looking trace with a breached metric means you have an uninstrumented step.
2. Online evals. Metrics/traces/logs confirm the request was up, fast, and cheap — none of them scores correctness. A confident hallucination is a 200 OK; only scoring the output (LLM-judge + grounding) reveals it.
3. Tiered sampling: keep metrics at 100% (cheap), head-sample detailed traces for normal traffic (10–20%), and tail-sample to keep 100% of slow/errored traces — the exact ones you'd open. You drop the boring traces, not the useful ones.
4. Because a single slow request (or a brief blip) shouldn't page anyone — and a slow, steady breach that will blow your monthly budget should. Burn-rate over multiple windows distinguishes a real incident from noise, which is what keeps alerts trustworthy and stops fatigue.

• “200 OK means it worked.” → It means the request completed. The answer can still be wrong, unsafe, or hallucinated — a perfect 200. Only an eval signal judges the output.
• “Logs are enough.” → Logs are per-event text; without a trace stitching them with a shared id you can't see the shape of a request or which hop was slow. The trace is the spine.
• “I'll just scale my APM / uptime monitor.” → Uptime and CPU don't capture tokens, cost, TTFT, or quality. You need LLM-aware signals (OTel GenAI conventions), not just ‘is the box up?’.
• “Trace everything at 100%.” → That's a cost and storage bomb. Metrics go 100%; traces are sampled (with tail-sampling for the slow/error ones). Visibility ≠ capturing every byte.
• “Just dump prompts into the dashboard.” → Prompts and outputs are full of PII. Redact before storing, use differential logging, and hash for pattern analysis — observability is a governance surface (L236/L237).
• “Offline evals in CI are enough.” → They catch the cases you imagined. Online evals on live traffic catch the drift and edge cases you didn't — production is the only test set that's fully real.
• “More alerts = safer.” → Noisy alerts get ignored. Alert on SLO / error-budget burn, tier by severity (safety pages, drift tickets), and protect the signal-to-noise of your pager.

• Four signals, not one: metrics (something's wrong), traces (where & why), logs (what the model saw), evals (was it any good). You need all four.

• Instrument once with OpenTelemetry + the GenAI semantic conventions — vendor-neutral spans with gen_ai.* attributes (model, tokens, cost, TTFT) so you can switch backends freely.

• The trace is the spine: propagate context across every hop (W3C traceparent) so six services become one waterfall. An uninstrumented hop is a blind spot — you can't debug what you didn't instrument.

• Emit LLM-specific signals (tokens, cost, TTFT, pinned versions, finish_reason); metrics at 100%, traces sampled with tail-sampling for slow/error.

• Logs are a governance surface: redact PII before storing, use differential logging and hashing (ties to L237 audit).

• Make quality a live signal: online evals score sampled live traffic async — the only thing that catches a silent regression, since a hallucination is a 200 OK. Watch drift.

• Alert on SLO / error-budget burn, tiered by severity (safety pages, drift tickets) — and close the loop: bad trace → eval case → fix → new version → re-deploy.

• Next — L250: Vision & Image Understanding (Multimodal AI Engineering). That completes Deploying & Scaling — you can now build, ship, scale, and see a production LLM app end to end. The next section opens a new capability: models that don't just read text, but see.