Offline vs Online Evaluation

Introduction

You're convinced you need evals (The Eval-Driven Development Mindset) and you respect why they're hard (Why Evaluating LLMs Is Uniquely Hard). Now a practical question: when do you actually run them? There are two fundamentally different moments — before you ship, on a dataset you control, and after you ship, on real users — and they are not interchangeable.

They answer different questions, catch different failures, and have opposite strengths. Doing only the first lulls you into false confidence; doing only the second means you find every problem the hard way, in production. This lesson maps the two modes — offline and online — and, more importantly, the loop that connects them into a system that keeps getting smarter.

An infographic titled 'Offline vs Online Evaluation' showing the two complementary modes of evaluating an LLM system and the loop between them. On the left, OFFLINE EVALUATION runs before deployment: a fixed, curated golden dataset feeds a continuous-integration gate that produces a pass or fail, blocking a release if a known case regresses; its attributes are listed as fixed and reproducible, you control everything, ground truth is available because you labeled it, and it catches KNOWN failure modes — described as your pre-flight check against known cases. On the right, ONLINE EVALUATION runs in production: real, messy, novel, sometimes adversarial user traffic flows into continuous monitoring using reference-free scorers, implicit signals like thumbs-down rate and abandonment, and runtime guardrails; its attributes are real distribution, usually no answer key, and it catches NOVEL failures and drift — described as your continuous instrument on reality. Between them a cycle is drawn: a top arrow labeled ship carries a validated change from offline to online, and a bottom arrow labeled production failures become new cases carries real incidents back from online into the offline set, closing the loop so the eval set keeps improving. A band warns that a great offline score is necessary but not sufficient — there is no universal formula that predicts online performance from offline metrics, because of distribution shift, an unrepresentative test set, and evolving user behavior — and that you roll changes out safely along a ladder from shadow to canary to A/B testing, watching guardrails, not just the goal metric, and remembering that non-determinism inflates variance so A/B tests need far larger samples. Three cards summarize: offline gates known cases before deploy; online instruments reality without a ground truth via reference-free evals, feedback, and guardrails; and the loop turns every production failure into a regression test. The takeaway banner reads: offline is your pre-flight check against known cases and online is your continuous instrument on reality — you need both, and every online surprise should become tomorrow's offline test.

The Two Modes

The distinction is about when you evaluate and what data you evaluate on:

Offline evaluation happens before deployment, on a fixed, curated dataset (a “golden set”). You run it in CI, it's fully reproducible, you control every variable, and — crucially — you have ground truth, because you wrote or labeled the expected answers. It answers: “Did this change pass the bar on the cases we already know about?” It is your pre-flight check.

Online evaluation happens after deployment, on real production traffic — messy, novel, sometimes adversarial inputs from actual users. It runs continuously, on a sample of live requests, and you usually have no answer key (nobody labels production in real time). It answers: “Is it actually working for real people, right now?” It is your continuous instrument on reality.

	Offline	Online
When	pre-deploy (CI gate)	in production, continuous
Data	fixed, curated golden set	real, live, unpredictable traffic
Ground truth	yes — you labeled it	usually none
Catches	known failure modes, regressions	novel failures, drift, real-load issues
Risk	zero — no user sees it	real — failures already reached someone

The one-liner to remember: offline is your pre-flight check against known cases; online is your continuous instrument on reality. You need both.

See It: Where Does Each Failure Get Caught?

The fastest way to internalize the division of labor is to sort real failures into the mode that would catch them. For each scenario below, make the call — offline, online, or both — then check your reasoning.

The pattern: known, reproducible, pre-deploy → offline. Novel, drifting, load- or distribution-dependent → online. And anything that's both a known threat and needs to be stopped live (like a jailbreak) belongs in both — tested offline, enforced online.

Offline Is Necessary — but Not Sufficient

Here's the trap that catches teams who love their offline suite: a great offline score does not guarantee online success. Researchers have tried to predict production performance from offline metrics and found no universal formula — the correlation is real but loose. Your model can ace the golden set and still struggle the moment real users arrive.

Why the gap? Three reasons, all flavors of distribution shift:

Your test set isn't fully representative — you wrote it, so it reflects the cases you thought of, not the ones users actually send.
User behavior evolves — new slang, new use cases, new ways of phrasing things your set never saw.
Production has a long tail of edge cases no curated set of 50 or 500 can cover.

So offline is the gate you must pass, not the proof you're done. Treat a green offline run as “safe to expose to reality,” never as “verified correct in the wild.”

The Online Problem: No Answer Key

Online's hardest constraint is the missing ground truth: you can't compute accuracy on live traffic because nobody labeled it. So how do you measure quality in production? Four tools, none of which need an answer key:

Reference-free evaluators. A grader that judges an output on its own terms — an LLM-as-judge against a rubric, or a code check (valid JSON, required fields, length, no PII). No expected answer required, so it runs on anything.
Implicit feedback. Users tell you constantly without being asked: thumbs-down rate, regeneration clicks, session abandonment, copy actions. These are noisy but free and real-time.
Explicit feedback. The occasional rating, report, or correction — sparse but high-signal.
Guardrails as runtime enforcement. A score reports; a guardrail acts. The same check that grades offline can run inline to block, flag, or fall back on a bad output before the user sees it — because a number alone never stopped anything.

The elegant part: the very same scorer you wrote for offline can run online. Define your grader once; use it as a CI gate and as a live guard. Here's exactly that — one reference-free, Claude-powered grader, two worlds:

from anthropic import Anthropic
client = Anthropic()

# A REFERENCE-FREE scorer: it judges quality WITHOUT a ground-truth answer, so the
# SAME function works offline (on your golden set) AND online (on live traffic, where
# there is no answer key). Define the grader once; use it in both worlds.
def is_grounded(question: str, answer: str, context: str) -> bool:
    verdict = client.messages.create(
        model="claude-opus-4-8", max_tokens=5,
        system="Reply only PASS or FAIL. PASS iff every claim in the answer is "
               "supported by the context — no outside knowledge.",
        messages=[{"role": "user",
                   "content": f"CONTEXT:\n{context}\n\nQ: {question}\nA: {answer}"}])
    return verdict.content[0].text.strip().upper().startswith("PASS")

# OFFLINE — in CI, gate the release on your curated set (you HAVE the contexts):
offline_score = sum(is_grounded(**case) for case in GOLDEN_SET) / len(GOLDEN_SET)
assert offline_score >= 0.95, "regression — block the deploy"

# ONLINE — sample live traffic and guard/alert when the same score fails:
if not is_grounded(user_q, live_answer, retrieved_context):   # runtime guardrail
    log_for_review(user_q, live_answer)                       # -> tomorrow's offline case
    serve_fallback()                                          # a score alone won't stop it

One function, two jobs: it gates the deploy offline and guards the user online — and the cases it flags live flow straight back into the golden set.

Rolling Out Safely: Shadow → Canary → A/B

Online evaluation isn't a single switch — it's a ladder of increasing exposure, each rung limiting blast radius:

Shadow. Run the new version on real traffic but don't serve its output — compare silently against the current version. Zero user risk; you see how it'd behave on reality before anyone depends on it.
Canary. Serve the new version to a small slice of traffic (say 1–5%) with automatic rollback wired to explicit thresholds — a p99-latency spike or a refusal-rate jump trips it instantly. Limits the damage if something slipped through.
A/B test. Split real users between variants to measure which actually wins on your metrics. Two cautions specific to LLMs: non-determinism inflates variance, so you need much larger samples than a normal A/B — often tens of thousands of sessions per arm to reach significance; and always watch your guardrails, because a variant can improve the goal metric while quietly regressing safety or cost.

Notice the through-line: each rung is a way to evaluate on reality while keeping the downside small.

Close the Loop: Production → Dataset

Here's where offline and online stop being two separate activities and become one engine. The most valuable thing online evaluation produces isn't a dashboard — it's new test cases.

Every time production surfaces a real failure — a hallucination, an angry thumbs-down, a novel input the model fumbled — that trace becomes, in one click, a new row in your offline golden set, and a regression test that runs in CI forever after. The unknown-unknown that online caught is now a known case that offline guarantees. Do this continuously and your eval set grows into something better than anything you could have written upfront — because it's made of the failures reality actually produced.

That's the whole system: offline gates → ship → online instruments → failures feed back → offline grows. Online finds the unknowns; offline locks them down; the loop compounds. (The flywheel mechanics — error analysis, taxonomies, regression gates — are the heart of the upcoming Error Analysis & the Eval Flywheel section.)

🧪 Try It Yourself

Map the two modes onto something you actually run (a chatbot, a RAG answer, an agent). Write down four things:

One failure your offline set would catch — a known case you can encode as a golden test today.
One failure only production would reveal — a drift or novel input your set can't anticipate.
One implicit signal you could log right now — thumbs, regenerations, abandonment, a follow-up “no, I meant…”. (If you can't name one, that's your highest-leverage gap.)
The one-click loop — where would a flagged production trace land so it becomes tomorrow's offline test?

If you only had answers for #1, you've been flying with a pre-flight check and no instruments. If only #2, you're flying on instruments with no pre-flight check. The lesson is that you need the full loop.

Mental-Model Corrections

“A strong offline score means I'm done.” No — offline is necessary but not sufficient. There's no formula that predicts online from offline; a green CI run means “safe to expose to reality,” not “verified in the wild.”
“Online evaluation needs ground truth, which I don't have, so I can't do it.” You can — reference-free evaluators, implicit feedback (thumbs/abandonment), and guardrails all work without an answer key.
“I'll A/B test it like any other feature.” LLM non-determinism inflates variance — you need far larger samples, and you must watch guardrails (safety, cost), not just the headline metric.
“Online evaluation is just monitoring dashboards.” It's active evaluation — reference-free scoring, runtime guardrails that block bad outputs, and safe rollout (shadow → canary → A/B), not passive charts.
“Offline and online are separate systems.” They're one loop: the same scorers run in both, and every online failure becomes an offline regression test. Designing them apart wastes the compounding.

Key Takeaways

Two modes, two questions. Offline (pre-deploy, fixed golden set, ground truth) asks “did this pass our known bar?”; online (production, real traffic, usually no labels) asks “is it working for real people now?” Offline = pre-flight check; online = instrument on reality.
Offline is necessary but not sufficient. No formula predicts online performance from offline scores — distribution shift, unrepresentative sets, and evolving users open a gap. A green CI run means safe to expose, not verified.
Online works without an answer key via reference-free evaluators, implicit/explicit feedback, and guardrails that act (block/flag/fall back), not just report. The same scorer can run offline and online.
Roll out on a ladder: shadow → canary → A/B, each limiting blast radius. Watch guardrails, not just the goal metric, and budget big samples because non-determinism inflates variance.
Close the loop. Every production failure becomes, in one click, a new offline regression test. Online finds the unknowns; offline locks them down; the loop makes your eval set better than anything you could write upfront.