Online Evals & Guardrail Metrics

You can trace your system (L177 (Tracing LLM & Agent Calls)), you've picked a tool to hold it all (L179 (Observability Tools)), and you've validated evals on a gold set that gates every change in CI (L175 (Catching Regressions Before They Ship)). But the gold set only contains the failures you already know about. Production is where the unknown failures live — the weird inputs, the gradual drift, the regression no test caught.

This lesson adds the two layers that run on live traffic: online evals (score real production responses to measure quality) and guardrails (inline checks that block what must never happen). They sound similar but they are opposite tools for opposite jobs, and the single most important skill is knowing which to use for each check. Get it wrong in one direction and harmful outputs reach users while you watch a dashboard; get it wrong in the other and you refuse 15% of perfectly good answers. Let's draw the line precisely.

By 2026 the consensus is that evaluation runs at three points in the lifecycle, and they're complementary — not competitors:

1. Offline — against a fixed golden dataset, in development and as a CI gate before any change ships (L173–L175). Controlled, repeatable, with ground-truth labels.
2. Online — against live production traffic, after responses are sent. Sampled, asynchronous, reference-free. This catches “the edge cases outside your test set, gradual drift, and regressions introduced by any change to the live system.”
3. Guardrails — inline in the request/response path, synchronously, blocking or modifying a response before the user sees it.

Offline answers “is this change safe to ship?” Online answers “how is it actually doing in the wild?” Guardrails answer “did this specific response cross a hard line?” This lesson is the bottom two — the production layers — and the rest of it is really one question: for any given check, does it belong in an online eval or a guardrail?

An online eval scores your system's actual production responses. Three properties define it:

• Asynchronous, off the hot path. Scoring runs after the response is returned to the user, so it never adds latency to the user-facing request.
• Reference-free. You don't have a gold answer for a random production query, so you score with: heuristics (JSON valid? sources cited? length in bounds?), reused guardrail signals (PII/toxicity detections are quality signals too), and an LLM-as-judge for open-ended qualities (helpfulness, faithfulness) — the reference-free judge from L166–L169.
• Sampled. A judge call is another model call — “scoring 100% of traffic is rarely worth it.”

Why bother, if you already gate in CI? Because production surprises you. A model-provider update, a traffic-mix shift, a prompt edit that helped your test cases but hurt a segment you didn't sample — these are invisible to a fixed offline set and only show up when you measure the live stream. Online evals are how you “find out before your customers do.” Run them at the gateway, which already sees every request, response, model, route, latency, and cost — the natural place to attach a quality score.

Because scoring costs money and time, how you sample and how you read the result are the whole craft:

• Sample a small fraction, target the risk. A common pattern: ~20% on high-risk routes, ~2% baseline elsewhere. You're producing a statistical estimate, not a census.
• Always report the sample size and a confidence interval. A quality score from 12 sampled responses on a low-volume route is noise — “be skeptical of moves on low-volume slices.” A drop only counts as a regression if it's statistically significant (the rigor from L165 (Statistical Rigor: Sample Size & Confidence)).
• Slice everything. Tag each score with route, model, prompt_version, judge_model. When quality drops, the slices tell you what changed — and tagging judge_model stops “the evaluator changed” from masquerading as a real quality drop.
• Trend, don't trust. The judge is a noisy estimator with biases (length, self-preference). Calibrate it to humans (L169 (Judge Biases & How to Calibrate)) and watch the trend over time, not any single verdict.

Here's the gateway pattern:

The payoff of slicing and sampling well: when a change moves quality down on a slice, you get an alert (or even an auto-rollback of that route to the prior model/prompt) — before customers feel it. The online eval closes the loop from production back to your deploy decisions.

A guardrail is the opposite tool. It runs synchronously, inline in the request/response path, and it can block or modify a response before it reaches the user — an input gate (before the model: injection, PII, off-topic) and an output gate (after the model: PII leak, toxicity, format). You met the mechanics of guardrails in the agent-safety lessons; here the point is when and the metrics.

The defining property is that a guardrail is an enforcement mechanism, not a measurement. And the killer rule comes from a real production lament:

“We'll track PII-exposure rate in our dashboards.” Cool — you just exposed customer data 47 times last week and have a nice chart showing it. If something must not happen, guardrail it. Don't measure how often you failed.

Guardrails don't have to be slow, either. The trick: run the output guardrail in parallel with the LLM call — “fire both off, wait for both to return.” Since the LLM dominates the latency, the guardrail's cost becomes negligible. (Input guardrails you must await, but they're fast, deterministic checks.) Keep guardrails fast and deterministic — they're a millisecond-budget safety net, not a place for a slow, fuzzy LLM judge.

Here's the decision, made concrete. You have five checks — three safety (PII, toxicity, injection) and two quality (helpfulness, faithfulness). Place each one as a guardrail (inline, blocks) or an online eval (async, samples, measures), tune the sampling rate, and watch the consequences. Try the two classic mistakes — a safety check as an online eval, a fuzzy quality judge as a guardrail — and read what each one costs.

The console makes the asymmetry visceral. Move a safety check to “online eval” and the harmful-to-users counter climbs — you're now charting breaches instead of stopping them. Move a quality judge to “guardrail” and the over-block rate jumps to ~15% while p95 latency balloons past a second. Put each where it belongs and everything stays green: fast inline blocks, quality measured calmly on a sample.

Distilled to a rule you can apply to any check:

If something must not happen → guardrail it (inline, synchronous, blocks).
If you want to know how good it is → eval it (async, sampled, measures).

The two failure modes are mirror images, and both are common:

• Safety as an online eval (async where you needed inline) → harmful outputs reach users. You only sample-measure the breach after it happened. “If something must not happen, don't measure how often you failed.” PII, toxicity, injection, unauthorized commitments → guardrail.
• Quality as a guardrail (inline where you needed async) → over-blocking + latency. “Block all responses where faithfulness < 0.8” sounds safe until “15% of your responses return ‘I cannot help with that’ for totally reasonable questions — users hate it,” and a slow judge in the request path blows up p99 latency. Helpfulness, faithfulness, tone → online eval. “Quality metrics are for monitoring and improvement, not real-time blocking.”

Here's the decision as code — note the output guardrail runs in parallel with the model, and the online eval is spawned (never awaited):

One nuance: the line isn't always obvious. A hard, cheap-to-check quality rule (valid JSON, required citation present) can be a guardrail — it's deterministic and must-pass. A fuzzy, judgment quality (is this helpful? is the tone right?) must be an online eval. The test: can you state it as a crisp must/never, checkable in milliseconds? Then guardrail. Otherwise, measure.

Guardrails block, but you still have to measure the guardrails themselves — a guardrail you don't monitor is a guardrail you don't trust:

• Block rate — what fraction of traffic each guard blocks/modifies. A sudden spike means an attack or a broken upstream; a drop to zero means the guard silently failed (fail-open).
• False-positive / over-block rate — how often the guard blocks a legitimate request (the “I cannot help” on a reasonable question). This is the guard's cost to good users; track it and tune the threshold down when it's too high (the calibration from the moderation lessons).
• Added latency — p50/p95/p99 the guard contributes. Near-zero if you run output guards in parallel; watch for a slow guard creeping onto the hot path.
• Coverage — which categories you actually guard (PII? toxicity? injection? jailbreak?) and what's still unguarded.

And the connection back to online evals: guardrail signals are also quality signals. The PII/toxicity/injection detections your guards produce feed the online-eval picture alongside heuristics and judges — one quality estimate, assembled from cheap deterministic checks and sampled judges. Cost, latency, and errors are nearly free to measure; quality is the expensive one — which is exactly why you sample it.

Stand up the production layer for one feature this week:

1. List your checks and place each one. For every check ask: must this never happen? → guardrail. Do I just want to know how good it is? → online eval. Write the list; you'll likely find one mis-placed.
2. Add one guardrail for your scariest must-never (PII leak, toxicity) — and run the output check in parallel with the model so it costs ~0ms. Track its block rate and over-block rate.
3. Add one online eval. Sample ~2–20% of traffic (more on high-risk routes), score it async with a reference-free claude-opus-4-8 judge for helpfulness or faithfulness, and log the score with its route, model, prompt_version, and sample size.
4. Put a CI on the number. Plot quality over time with its confidence interval; alert only on significant drops, and ignore low-volume slices.
5. Wire one alert (or auto-rollback) for when a slice's quality drops significantly after a change.

Do this and you have eyes on production: hard lines enforced, soft quality trended — the two things a fixed offline set can never give you.

• “Online evals and guardrails are the same thing.” Opposite tools. Guardrail = inline, synchronous, blocks (enforcement). Online eval = async, sampled, measures (monitoring).
• “I'll monitor our PII-exposure rate.” No — if it must not happen, BLOCK it. A dashboard of breaches is “a nice chart” of customer-data leaks. Don't measure how often you failed; guardrail it.
• “Block every response with faithfulness < 0.8.” You'll over-refuse ~15% of good answers (“I cannot help”). Quality is for monitoring, not blocking — make it an online eval.
• “Guardrails add too much latency.” Run the output guardrail in parallel with the LLM (fire both, await both) — the model dominates, so the guard adds ~0ms. Input guards are fast, deterministic checks.
• “Score 100% of production.” A judge call costs money and time — sample (2–20%, target high-risk routes). It's a statistical estimate: report it with a sample size + CI.
• “The online quality score is the truth.” It's a noisy judge estimate — biased and inconsistent. Calibrate to humans, trend it, and don't act on low-volume slices.

• Evals run at three points: offline (gold set, CI gate — L175), online (live traffic, sampled, async), and guardrails (inline, blocking). This lesson is the production two.
• Online evals MEASURE quality on live traffic: async (off the hot path), reference-free (heuristics + reused guardrail signals + an LLM judge), and sampled (~2–20%, target high-risk). Report every score with sample size + a CI, slice by route/model/prompt_version, and trend it — the judge is a noisy estimator, not a verdict.
• Guardrails BLOCK what must not happen: inline + synchronous; run output guards in parallel with the LLM to stay fast. They're enforcement, not measurement.
• The placement rule: if it must not happen → guardrail it; if you want to know how good it is → eval it. Safety-as-online-eval ships harm (“don't measure how often you failed”); quality-as-guardrail over-refuses ~15% of good answers + blows up latency.
• Measure the guardrails too: block rate, false-positive/over-block rate, added latency (p99), and coverage. Guardrail signals are also quality signals.
• Next: with quality being measured continuously, watch the trend for drift & degradation (L181 (Detecting Drift & Degradation)).