Reference-Based vs Reference-Free

Introduction

Pull back from the individual metrics for a moment, because there's an organizing idea that matters more than any single one: every evaluation metric belongs to one of two camps, and knowing which camp you're in decides what you can even measure.

Reference-based evaluation compares your output to a known-correct gold answer. Reference-free evaluation judges the output on its own — against the input, the retrieved context, or a rubric — with no gold answer at all. Everything you've met so far slots in: exact match and similarity are reference-based; perplexity and faithfulness are reference-free. This split is the bridge from “grading a test set” to “evaluating a live product” — and it contains one hard truth that surprises most teams: you cannot run your favorite reference-based metrics on production traffic.

An infographic titled 'Reference-Based vs Reference-Free' showing the fundamental split that organizes every evaluation metric: whether you compare the output to a gold answer or judge it on its own. On the left, REFERENCE-BASED evaluation scores the output against a known-correct gold answer or label, using exact match, BLEU and ROUGE, F1, or embedding similarity; its strengths are that it is objective, repeatable, and automatable, making it the right tool for offline benchmarks and regression tests; its limits are that it needs expensive hand-labeled references, assumes a single correct answer, penalizes valid alternative phrasings, and the references go stale. A red wall warns that you cannot run reference-based metrics on production traffic, because ten thousand live user queries a day have no gold answers, so reference-based is an offline-only tool. On the right, REFERENCE-FREE evaluation scores the output with no gold answer at all, judging it against the input, the retrieved context, policies, or a rubric; the family includes faithfulness or groundedness, meaning every claim is supported by the context, plus answer relevance, context relevance, safety and toxicity, format and schema validity, perplexity, self-consistency, and an LLM judge on a rubric, none of which need a label, so they run on every live response. The canonical example is the RAG triad of three reference-free scores: faithfulness that the answer is grounded in the retrieved context, answer relevance that the answer addresses the query, and context relevance that the retrieved context is relevant, all computable on production traffic with zero gold answers. A warning panel flags the catch in reference-free evaluation: groundedness is not correctness, because a grounded answer is faithful to the context yet still wrong if the context itself is wrong, and reference-free judges carry optimistic and cynical biases. A band gives the resolution: the answer is to use both, mapping reference-based to offline curated regression gates and reference-free to online production monitoring, combining them in a hybrid pipeline. The takeaway banner reads: reference-based needs a gold answer and is objective but offline-only and one-answer, reference-free needs only the input, context, or a rubric and is the only thing that works on production traffic, so pick the camp by whether you have references and use both where each fits.

Reference-Based: Compare to a Gold Answer

This is the camp we've mostly lived in. A reference-based metric needs a trusted reference — a hand-labeled correct answer — and scores how close the output gets to it: exact match, BLEU/ROUGE, F1, embedding similarity. All of them are some flavor of “how well does the output match the gold?”

# REFERENCE-BASED: compare the output to a known-correct GOLD answer.
GOLD = {"q123": "Refunds take 3-5 business days."}          # a hand-labeled test set
def reference_based(answer, qid):
    return answer == GOLD[qid]                              # or BLEU / F1 / similarity

# ...but a LIVE production query has no gold answer:
live_query = "How long does a refund take?"                # which qid? which gold? none.
# reference_based(answer, ???)   <- impossible. Reference-based is an OFFLINE tool.

Its strengths are real: it's objective, repeatable, and automatable — the same input always yields the same score, with no judge to bias it. That makes it the ideal tool for offline benchmarks and regression tests: curate a set with gold answers once, and you can re-run it for free on every change forever. Its weaknesses are equally real: you need expensive hand-labeled references, it assumes there's one right answer (so it penalizes valid alternatives, as the similarity lesson showed), and references go stale. But there's a deeper problem coming.

The Wall: You Can't Reference-Based Production

Here's the truth that quietly breaks a lot of evaluation plans. In production, you have no gold answers. Ten thousand real users a day type ten thousand queries nobody has ever written a correct answer for. There is nothing to compare against — so exact match, BLEU, F1, and similarity are simply not computable on live traffic.

This is why reference-based evaluation is fundamentally an offline tool. It lives on your curated test sets, gating changes before they ship (the offline half of Offline vs Online Evaluation). The moment you face the online world — monitoring real responses, catching a regression in production, flagging a hallucination on a query you've never seen — reference-based hits a wall. You need a way to evaluate an output without a reference at all.

Reference-Free: Judge the Output on Its Own

Reference-free evaluation scores an output with no gold answer. Instead of “does it match the reference?”, it asks “is this output good — given the input, the context, and what we care about?” The evidence is the input, the retrieved context, a policy, or a rubric — never a labeled target. It's a whole family:

Faithfulness / groundedness — is every claim supported by the retrieved context? (the anti-hallucination check)
Relevance — does the answer actually address the question asked?
Safety / toxicity / bias — does it comply with policy? (checked against rules, not a gold answer)
Format / schema validity — is it valid JSON, the right length, the required fields? (cheap, deterministic, reference-free)
Perplexity — the intrinsic surprise measure from the last concepts lesson.
Self-consistency and an LLM-as-a-judge on a rubric — score quality directly.

The magic property: none of these need a label, so every one of them runs on live production traffic. This is what lets you monitor a system you can't pre-label — which is most real systems.

The RAG Triad: Reference-Free in Action

The cleanest example of reference-free done right is the RAG Triad — three scores that together tell you whether a retrieval-augmented answer is trustworthy, and not one of them needs a gold answer:

Faithfulness (groundedness) — is the answer grounded in the retrieved context, with no invented facts?
Answer relevance — does the answer address the user's question?
Context relevance — did retrieval actually fetch context relevant to the query?

All three are computable from just the query, the context, and the answer — which you always have, even on live traffic. Here's the faithfulness check, the heart of the triad:

# REFERENCE-FREE: grade with NO gold answer -- only the query, the context, the answer.
from anthropic import Anthropic
client = Anthropic()

def grounded(answer, context):     # FAITHFULNESS: is every claim supported by the context?
    r = client.messages.create(model="claude-opus-4-8", max_tokens=10,
        system="Reply YES only if EVERY claim in the answer is supported by the context.",
        messages=[{"role": "user", "content": f"CONTEXT:\n{context}\n\nANSWER:\n{answer}"}])
    return r.content[0].text.strip().startswith("YES")

# Runs on EVERY live response -- no labels. (The RAG triad adds answer- & context-relevance.)
# The catch: "grounded" means true-to-the-CONTEXT, not true. Wrong context -> grounded but wrong.

Run that on every production response and you have a live hallucination monitor — no labels, no curated set. (The other two triad scores follow the same shape, asking a judge about relevance instead of grounding.) This is reference-free evaluation earning its place as the backbone of production monitoring.

See It: Which Camp Can Grade This?

Feel the split across real situations. For each scenario below, see how reference-based and reference-free each fare — and watch reference-based hit its wall the moment you leave the curated test set.

Interactive: pick a scenario (offline regression test with gold answers / live production with no labels / an open-ended writing task / a RAG answer) and compare reference-based vs reference-free side by side — each shows whether it CAN grade the case, its metrics, and a verdict, plus a recommendation. Reference-based wins on the curated offline set but is impossible on production and open-ended tasks, where only reference-free works; the RAG scenario flags that grounded is not the same as correct.

The arc is unmistakable: with a curated offline set, reference-based is the better tool — objective and free to re-run. The instant you step into production or open-ended work, the gold answers vanish and reference-free is the only thing left standing. The metric you can use is dictated by whether a reference exists.

The Catch: Groundedness ≠ Correctness

Reference-free is powerful, but it measures something subtly different from truth, and you must keep the distinction sharp. The biggest trap is groundedness ≠ correctness.

Faithfulness checks that every claim in the answer is supported by the retrieved context — but it says nothing about whether that context is right. If retrieval pulls a stale or wrong document, an answer that faithfully repeats it is perfectly grounded and completely wrong. Groundedness is adherence to the source, not truth. (That's also why context relevance is part of the triad — to catch the bad-context case.)

The second catch: reference-free judges have biases. Studies find both optimistic bias (waving through wrong answers as correct) and cynical bias (failing correct ones), so an LLM-judge score is a strong signal, not gospel — it needs calibration (the next section). The lesson: know exactly what each reference-free score measures. “Grounded,” “relevant,” and “correct” are three different questions — don't let a high groundedness score lull you into assuming the answer is true.

The Answer Is Both: Hybrid Eval

This isn't a contest with a winner — it's a division of labor, and mature teams use both. The mapping is clean:

Offline, with curated references → reference-based. Objective, free to re-run, perfect as a regression gate before you ship.
Online, in production, no labels → reference-free. Faithfulness, relevance, safety, and judge scores running on live traffic, catching hallucinations and drift the offline suite can't see.

A healthy evaluation pipeline is hybrid: a reference-based regression suite guarding your known cases, plus reference-free monitors watching the unknown ones in production — each used exactly where it fits. The two camps cover each other's blind spots: reference-based gives you objectivity where you have ground truth; reference-free gives you reach where you don't. Reach for the camp the situation allows, and over a whole system, you'll use both.

🧪 Try It Yourself

Map your own feature onto the split — it takes three questions:

Do you have gold answers? For your test set, probably yes → reference-based is available. For your live production traffic, almost certainly no → you'll need reference-free. (Most systems are both, in different places.)
Write one of each. A reference-based check for your test set (output == gold, or F1). And a reference-free check you could run on a live response (e.g., “is every claim supported by the retrieved context?” — faithfulness).
Run the groundedness ≠ correctness test. Imagine your retriever returned a wrong document. Would your reference-free faithfulness check still pass the answer? (It would — that's the catch.) What would you add to catch it? (Context relevance, or a correctness check against a trusted source.)

If you've only ever evaluated on a test set, question 1 just revealed your blind spot: you have no eyes on production. Reference-free is how you open them.

Mental-Model Corrections

“I'll just run BLEU / F1 / exact match in production.” You can't — production has no gold answers to compare against. Those are offline tools. Use reference-free on live traffic.
“Reference-free means using an LLM judge.” A judge is one kind — but reference-free also includes format/schema checks, safety filters, and perplexity, many of which are cheap and deterministic. Not everything reference-free is expensive.
“If the answer is faithful/grounded, it's correct.” No — groundedness ≠ correctness. A grounded answer is faithful to the context; if the context is wrong, the answer is wrong. Add context-relevance and (where possible) a truth check.
“Reference-based is outdated.” No — it's the best tool when you have references: objective, free to re-run, ideal for offline regression. Keep it for your curated sets.
“Pick one camp for the whole system.” No — it's a hybrid: reference-based gates offline; reference-free monitors online. Use each where the situation allows.
“Reference-free scores are objective truth.” They're strong signals with judge biases (optimistic and cynical). Calibrate them; don't treat a score as gospel.

Key Takeaways

Every metric is reference-based or reference-free, and which one you can use is dictated by whether a gold answer exists.
Reference-based (exact match, BLEU/ROUGE, F1, similarity) compares to a gold answer — objective and repeatable, but needs labels, assumes one answer, and is offline-only.
You can't reference-based production — live traffic has no gold answers, so reference-based hits a wall the moment you leave your curated test set.
Reference-free judges the output against the input, context, or a rubric — faithfulness/groundedness, relevance, safety, format, perplexity, judge. No labels needed → it runs on live traffic, which is why it powers production monitoring.
The RAG Triad (faithfulness · answer relevance · context relevance) is reference-free done right — three trust scores from just the query, context, and answer.
Mind the catch: groundedness ≠ correctness (faithful to a wrong context is still wrong), and judges have biases. Use both camps — reference-based to gate offline, reference-free to monitor online.