Building the Evaluation Harness

Introduction

You've built a real assistant: L2 gave it retrieve(), L3 gave it grounded, cited generate() with abstention. But here's an uncomfortable question — how do you know it's any good? And the harder one: when you change the chunk size, swap the reranker, or raise min_score, how do you know you helped instead of quietly breaking something? Right now the honest answer is you don't — you've been running a few queries and trusting your gut.

This lesson replaces vibes with measurement. You'll build an evaluation harness: a golden set of questions with known-good answers, scored automatically by RAGAS on four metrics that pinpoint exactly what's working and what isn't. By the end, make eval prints a scorecard, and you can prove a change helped, diagnose which component is weak, and gate every change in CI so a regression never reaches production.

Why this is the lesson that separates engineers from demo-builders. Anyone can build a RAG demo that looks impressive on three hand-picked questions. The thing companies actually pay for — and the thing interviewers probe — is "how do you know it works, and how do you keep it working as you change it?" Evaluation isn't a finishing touch; it's how you develop the system. Every knob you've set so far (min_score, contextual retrieval, top-k, the reranker) was a guess until now. The harness turns each into a measured decision.

We use the course's own stack: RAGAS for the metrics, with Claude as the judge LLM (an LLM scores the outputs of your LLM). Everything is runnable against the assistant you've built.

Hero infographic titled 'Building the Evaluation Harness' for the fourth lesson of the Production RAG Assistant capstone, on a white background. The deck says: stop trusting vibes — score the system with RAGAS, diagnose the failing component, and gate every change in CI. The centre shows an evaluation loop. A GOLDEN SET box (question plus reference answer, curated plus synthetic) feeds 'run YOUR retrieve() + generate()' which produces, per question, the user input, the retrieved contexts, the response, and the reference. Those flow into a RAGAS box (judge = Claude) that computes four metrics, drawn as two grouped pairs. The RETRIEVER pair: Context Recall and Context Precision. The GENERATOR pair: Faithfulness and Answer Relevancy. Each metric has a target — faithfulness greater than 0.9, answer relevancy greater than 0.85, context precision greater than 0.8, context recall greater than 0.8 — feeding a CI GATE diamond that reads PASS or FAIL versus baseline. A prominent DIAGNOSIS panel maps each low metric to the component to fix: low context recall means the retriever is missing chunks; low context precision means the context is noisy, add a reranker; low faithfulness means the generator is hallucinating beyond the context; low answer relevancy means the answer is off-topic. A small caution note reads: LLM-as-judge — calibrate against ~30 to 50 human labels (watch verbosity and position bias). Three summary cards along the bottom read: 'Golden set — curate real Q&A plus synthetic'; 'Four metrics localize failure — retriever vs generator'; 'Gate every change in CI against a baseline'. A family strip lists the five build lessons — Brief & Architecture, Ingestion & Hybrid Retrieval, Generation Citations & Guardrails, Building the Eval Harness, Deploy & Observe — with the fourth highlighted. Slate, sky, cyan, violet, amber, emerald accents; generous legible text.

Why Evaluation Is the Whole Game

Three traps kill RAG systems, and all three are eval problems:

"It feels better." You tweak the prompt, run your two favourite questions, and they look great. But "feels better" on a handful of cherry-picked queries is noise — you have no idea what it did to the other 95% of questions, including the ones it silently broke.
Silent regressions. You swap Sonnet for a cheaper model to save cost. Latency and bill drop, everything looks fine — and faithfulness quietly fell 8 points on exactly the questions you didn't re-check.
"Where do I even start?" A user reports a wrong answer. Was it retrieval (wrong chunks)? Generation (right chunks, bad answer)? Without component-level metrics you're left guessing whether to re-chunk, change the embedder, tune the prompt, or swap the model.

The principle: un-measured is unmanaged. You cannot improve — or even safely maintain — what you can't measure. An eval harness gives you a number that moves when you change something, so development becomes a tight loop: change → measure → keep or revert. That's eval-driven development, and it's the difference between engineering a system and poking at one.

The payoff cascades: the same harness that guides your iteration becomes your CI gate (block merges that regress) and your interview story ("here's how I measured it").

The Four Metrics — The RAG Triad, Localized

RAGAS scores a RAG answer with four metrics, and the magic is that they split cleanly by component — two grade the retriever, two grade the generator. That split is what turns "the answer is wrong" into "this part is wrong."

Retriever metrics (did we fetch the right context?):

Context Recall — of the information needed to answer (per the reference), how much made it into the retrieved chunks? Low recall = the retriever missed something. (Needs a reference answer.)
Context Precision — of the retrieved chunks, how many are actually relevant (and ranked high)? Low precision = the context is noisy, diluting the good chunks.

Generator metrics (given the context, did we answer well?):

Faithfulness — what fraction of the claims in the answer are supported by the retrieved context? Low faithfulness = the model is hallucinating beyond its sources. (This is the metric your citations from L3 make checkable.)
Answer Relevancy — is the answer on-topic and complete for the question, not evasive or padded? Low relevancy = the generation is off-target.

Metric	Grades	Asks	Needs reference?
Context Recall	Retriever	Did we fetch everything needed?	✅
Context Precision	Retriever	Is the fetched context clean?	✅
Faithfulness	Generator	Is the answer grounded in context?	❌
Answer Relevancy	Generator	Does it actually answer the question?	❌

Together they're the RAG triad (plus recall): a complete diagnostic picture. We'll set targets — faithfulness > 0.9, answer-relevancy > 0.85, context-precision > 0.8, context-recall > 0.8 — and treat anything below as a bug with a known address.

The Golden Set — Your Ground Truth

Evaluation needs a golden set: questions paired with a reference answer (and ideally the source the answer should come from). This is the single most valuable asset in the whole project — it's what every score is measured against. Build it two ways, together:

1. Curate real questions (start here). Hand-write 20–50 questions your users would actually ask, with correct reference answers. Pull them from real support tickets, search logs, or domain experts. Cover the hard cases on purpose: exact identifiers, paraphrases, multi-hop questions, and unanswerable ones (the right answer is "I don't know" — these test your L3 abstention). A small, honest golden set beats a big sloppy one.

2. Generate synthetically (scale up). RAGAS can build a test set from your documents — it reads your corpus and generates diverse questions (simple, multi-context, reasoning) with references, cutting curation time dramatically. Use it to expand coverage, then spot-check the results (synthetic data has errors too):

# src/rag/eval/make_golden.py  ->  writes eval/golden.jsonl
from langchain_community.document_loaders import DirectoryLoader
from ragas.testset import TestsetGenerator

from rag.eval.judge import judge_embeddings, judge_llm   # defined next section


def main(data_dir: str = "data", size: int = 30) -> None:
    docs = DirectoryLoader(data_dir, glob="**/*.md").load()
    generator = TestsetGenerator(llm=judge_llm, embedding_model=judge_embeddings)
    testset = generator.generate_with_langchain_docs(docs, testset_size=size)
    # columns: user_input (question), reference, reference_contexts
    testset.to_pandas().to_json("eval/golden.jsonl", orient="records", lines=True)
    print(f"wrote {size} synthetic examples -> eval/golden.jsonl (review them!)")

Store the golden set as eval/golden.jsonl — one JSON object per line: {"question": ..., "reference": ...}. Commit it to the repo. It's versioned ground truth; when it grows, your eval gets more trustworthy. (Keep the curated and synthetic examples together but tagged, so you can report scores on each.)

Wiring RAGAS to the Course Stack

RAGAS uses an LLM as the judge — to score faithfulness it asks a strong model "is each claim in this answer supported by this context?" On the course stack, that judge is Claude (here gen_model = claude-sonnet-4-6; use a strong model to judge), wrapped for RAGAS, with an embedding model for the similarity-based parts of relevancy. Define them once:

# src/rag/eval/judge.py
from langchain_anthropic import ChatAnthropic
from langchain_voyageai import VoyageAIEmbeddings
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.llms import LangchainLLMWrapper

from rag.config import settings

# Claude as the evaluator/judge LLM (use a strong model for judging).
judge_llm = LangchainLLMWrapper(
    ChatAnthropic(model=settings().gen_model, max_tokens=1024, api_key=settings().anthropic_api_key)
)
judge_embeddings = LangchainEmbeddingsWrapper(
    VoyageAIEmbeddings(model="voyage-3-large", api_key=settings().voyage_api_key)
)

A single RAG interaction is a SingleTurnSample with up to four fields — the user_input (question), the retrieved_contexts (what your retriever returned), the response (what your generator answered), and the reference (the golden answer). A set of these is an EvaluationDataset. The metric classes map straight to the four metrics:

# src/rag/eval/metrics.py
from ragas.metrics import (
    Faithfulness,
    ResponseRelevancy,                    # "answer relevancy"
    LLMContextPrecisionWithReference,     # "context precision"
    LLMContextRecall,                     # "context recall"
)

METRICS = [
    LLMContextRecall(),
    LLMContextPrecisionWithReference(),
    Faithfulness(),
    ResponseRelevancy(),
]

TARGETS = {                               # gate thresholds (tune on YOUR data)
    "context_recall": 0.80,
    "llm_context_precision_with_reference": 0.80,
    "faithfulness": 0.90,
    "answer_relevancy": 0.85,
}

Running the Eval — Score Your Own System

The crucial point: you evaluate your actual pipeline, end to end. For each golden question, call your retrieve() and your generate() (the real ones from L2/L3), capture what they produced, and hand RAGAS the result. This is what makes the score about your system, not a toy:

# src/rag/eval/run.py   ->  `python -m rag.eval.run`  (make eval)
import json

from ragas import EvaluationDataset, SingleTurnSample, evaluate

from rag.eval.judge import judge_embeddings, judge_llm
from rag.eval.metrics import METRICS
from rag.generate import generate
from rag.retrieve import retrieve


def build_dataset(golden: list[dict]) -> EvaluationDataset:
    samples = []
    for ex in golden:
        chunks = retrieve(ex["question"])           # L2: hybrid -> RRF -> rerank
        answer = generate(ex["question"], chunks)    # L3: grounded + cited + abstain
        samples.append(SingleTurnSample(
            user_input=ex["question"],
            retrieved_contexts=[c["text"] for c in chunks],
            response=answer.text,
            reference=ex["reference"],
        ))
    return EvaluationDataset(samples=samples)


def run() -> "pandas.DataFrame":
    golden = [json.loads(line) for line in open("eval/golden.jsonl")]
    dataset = build_dataset(golden)
    result = evaluate(dataset, metrics=METRICS, llm=judge_llm, embeddings=judge_embeddings)
    df = result.to_pandas()                          # per-question scores
    print(df[[m.name for m in METRICS]].mean())      # aggregate scorecard
    return df


if __name__ == "__main__":
    run()

result.to_pandas() gives you a per-question table (slice it to find which questions fail), and the column means are your aggregate scorecard. Now you have numbers — the next two sections are about acting on them.

Reading the Scores — Component Diagnosis

This is where the component split pays off. A single low metric tells you which part to fix — no more guessing. Internalize this map; it's the most useful table in the lesson:

If this is low…	The problem is the…	Because…	Fix (which lesson)
Context Recall	Retriever	the needed info never got retrieved	better chunking + contextual retrieval + raise top-k (L2)
Context Precision	Retriever	retrieved context is noisy/diluted	add/strengthen the reranker; lower top-k (L2)
Faithfulness	Generator	the answer claims things the context doesn't support	tighten the grounded prompt; raise `min_score`; also check precision (L3)
Answer Relevancy	Generator	the answer is off-topic or incomplete	improve generation; ensure recall covers the question (L3)

Worked example. Faithfulness is 0.72 (target 0.90) but context recall and precision are both fine (>0.85). The retriever is doing its job — the right context is there — so the bug is in generation: the model is adding unsupported claims. You'd tighten the system prompt and/or raise min_score; re-chunking would waste a day fixing a part that isn't broken. Without the split, you'd have guessed.

The order to read them: retrieval first (recall, then precision), then generation (faithfulness, then relevancy) — because generation can't beat its input. If recall is low, fix that before touching the prompt; a perfect generator on missing context still fails.

LLM-as-Judge — Trust, but Calibrate

The harness uses an LLM (Claude) to judge your LLM's outputs — which is powerful and scalable, but it comes with a caveat you must respect: the judge has biases of its own. Documented failure modes include verbosity bias (favouring longer answers regardless of quality), position bias (preferring the first/last option in a comparison), and self-enhancement bias. A judge that's systematically wrong gives you confident, precise, wrong numbers — the most dangerous kind.

So calibrate the judge against humans — once, deliberately:

Take 30–50 examples and have a human (you, or a domain expert) label them — is this answer faithful? relevant?
Run the RAGAS judge on the same examples and measure agreement with the human labels. Use a chance-corrected statistic like Cohen's κ, not raw accuracy (raw % overstates agreement).
If the judge disagrees with humans on clear-cut cases more than ~20% of the time, fix the judge (a stronger judge model, or a clearer metric prompt) before you trust its scores — then scale to the full set.

Think of it as testing your test. A calibrated judge is the foundation everything else stands on; an uncalibrated one means every downstream decision rests on noise. (This is also increasingly a compliance expectation — a judge that disagrees with humans in systematic ways is an audit finding, not a passing grade.)

Eval-Driven Development & CI Gating

Now use the harness the way pros do. Eval-driven development is a loop: form a hypothesis ("adding the reranker will lift precision"), make the change, run make eval, and keep it only if the numbers improved. Two high-value moves you can finally make with data:

A/B Contextual Retrieval (from L2): run the eval with the flag off, then on. Does context recall actually rise enough to justify the index-time cost? Now you know instead of assuming.
Choose min_score (from L3): sweep it and plot faithfulness vs. coverage. Higher min_score lifts faithfulness (fewer answers from weak chunks) but lowers coverage (more abstentions). Pick the knee of the curve for your risk tolerance — a measured threshold, not a guess.

Then gate it. Because prompts, model choice, and config are all code, every behavior-changing edit should run the eval suite in CI and be blocked on the scores before it can merge — this is how you stop silent regressions:

# src/rag/eval/gate.py   ->  exits non-zero if any metric is below target (fails CI)
import sys

from rag.eval.metrics import METRICS, TARGETS
from rag.eval.run import run


def gate() -> None:
    df = run()
    scores = {m.name: float(df[m.name].mean()) for m in METRICS}
    failed = {k: v for k, v in scores.items() if k in TARGETS and v < TARGETS[k]}
    for k, v in scores.items():
        mark = "OK " if k not in failed else "FAIL"
        print(f"  [{mark}] {k:<40} {v:.3f}  (target {TARGETS.get(k, 0):.2f})")
    if failed:
        sys.exit(f"\n✗ EVAL GATE FAILED: {failed}")
    print("\n✓ EVAL GATE PASSED")


if __name__ == "__main__":
    gate()

# .github/workflows/eval.yml  (run the gate on every PR)
name: rag-eval
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e ".[eval]"
      - run: python -m rag.eval.gate        # non-zero exit blocks the merge
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

That's the whole discipline: every change is measured, and regressions can't merge. Your assistant now improves on purpose.

✅ Definition of Done (this step)

Your eval/ module turns the assistant from "seems fine" into measured:

eval/golden.jsonl exists — 20–50 curated questions + references, plus synthetic ones from TestsetGenerator, committed to the repo.
Judge wired to the stack — Claude via LangchainLLMWrapper, an embeddings wrapper, and the four metric classes.
make eval runs your real retrieve() + generate() over the golden set and prints per-question + aggregate RAGAS scores.
You can diagnose — given a low metric, you know which component to fix (retriever vs. generator) and why.
Judge calibrated — agreement checked against ~30–50 human labels (κ), judge prompt/model fixed if it disagreed too often.
CI gate — rag.eval.gate exits non-zero below target; wired into a PR workflow so regressions can't merge.

If you can run make eval, read the scorecard, and say "precision is low, so I'll strengthen the reranker" — you're doing eval-driven development. Next, L5 ships it: deploy + observe.

See It — The Eval Harness Lab

This lab is make eval made playable. A five-question golden set is scored on the four RAGAS metrics against their targets, colour-coded by component — recall + precision (violet) grade the retriever, faithfulness + answer-relevancy (amber) grade the generator.

Start on the Naive preset. Every metric is red, the CI GATE reads FAIL, and the diagnosis card points at the retriever (precision is furthest below target — noisy context).
Turn on the Reranker and watch precision jump; turn on Contextual Retrieval and watch recall climb. Or just hit Production — every metric clears its target and the gate flips PASS. That's a fix you proved, not guessed.
Drag min_score up. Faithfulness rises (fewer answers from weak chunks) while coverage falls — the exact trade-off you'd tune on a real eval set.
Read the per-question table. Some questions fail while the aggregate passes — that's where you'd go hunting for the next improvement.

The Eval Harness Lab — the RAGAS harness from this lesson, made playable. A golden set of five Q&A is scored live on the four RAGAS metrics against their targets, with the retriever pair (context recall/precision) and generator pair (faithfulness/answer-relevancy) colour-coded by component. Change the real system config — toggle Contextual Retrieval and the Reranker, drag min_score, pick top_k — or snap to the Naive / Production preset, and watch every per-question row and the aggregate scorecard recompute, the CI GATE flip PASS/FAIL, and a diagnosis card name the failing COMPONENT and the exact fix. Start on Naive: the gate FAILs and the diagnosis points at the retriever (precision is lowest). Add the reranker and contextual retrieval (or hit Production) and watch every metric clear its target and the gate turn PASS. Then push min_score up and see faithfulness rise while coverage falls — the trade you tune on the eval set. Scores are a deterministic teaching model, not live judge calls.

Notice three things. One: a metric doesn't just say "bad" — it says which component to fix (the whole point of the split). Two: the reranker drives precision and contextual retrieval drives recall — levers map to metrics. Three: the gate turns a scorecard into a policy — below target, the change doesn't ship.

🧪 Try It Yourself

Reason these out, then check against the lab and the code.

1. Your eval shows context recall 0.62, but faithfulness and answer-relevancy are both > 0.9. Where's the bug, and what would you fix first — and what would be a waste of time to touch?

2. Why must you call your own retrieve() and generate() when building the evaluation dataset, rather than just scoring the golden answers directly?

3. A teammate reports faithfulness jumped from 0.88 to 0.96 after they raised min_score from 0.3 to 0.7 — "huge win!" What other number must you check before celebrating, and why?

4. Why calibrate the RAGAS judge against human labels with Cohen's κ instead of just trusting the scores (or using raw % agreement)?

5. Which two RAGAS metrics require a reference answer, and what does that imply for building your golden set?

Answers.

1. The bug is in the retriever — low context recall means the information needed to answer isn't making it into the chunks. Fix retrieval first (better chunking, contextual retrieval, higher top-k). Touching the prompt or the generator would be a waste — generation is already strong (faithfulness/relevancy high); it simply can't answer from context that was never retrieved. Generation can't beat its input.

2. Because you're evaluating your system, not the golden answers. The metrics need what your pipeline actually retrieved and generated — the retrieved_contexts and the response — to score whether your retriever found the right context and your generator stayed faithful to it. Scoring the golden answers alone would tell you nothing about your pipeline.

3. Coverage (how many questions still get answered vs. abstained). Raising min_score lifts faithfulness partly by refusing more questions — if coverage cratered, you didn't get more faithful, you got more silent. Faithfulness and coverage trade off; a 'win' on one that tanks the other is a regression in disguise.

4. Because the judge is itself an LLM with biases (verbosity, position, self-enhancement). A judge that systematically disagrees with humans gives confident, precise, wrong numbers. Cohen's κ is chance-corrected — raw % agreement overstates reliability (two judges agreeing by luck looks like skill). You calibrate to make sure the metric you're optimizing actually tracks human judgment.

5. Context Recall and Context Precision (with reference) need the reference. (Faithfulness needs only answer+context; answer-relevancy needs only question+answer.) Implication: your golden set must include reference answers — you can't measure retrieval completeness without knowing what a complete answer requires.

Mental-Model Corrections

"I'll add evaluation at the end." → Eval is how you develop the system, not a finishing touch. Without it, every change is a guess. Build the harness before you start tuning.
"It feels better on my test questions." → A handful of cherry-picked queries is noise. Only a golden set scored on metrics tells you if a change helped or quietly regressed.
"One overall quality score is enough." → A single score can't tell you what to fix. The four-metric split localizes failure to the retriever or the generator — that's the entire value.
"The LLM judge is objective." → The judge has biases (verbosity, position, self-enhancement). Calibrate it against human labels (Cohen's κ) before you trust it.
"A bigger generator will fix a low score." → If recall is low, no generator can help — the context is missing. Read retrieval metrics first. Generation can't beat its input.
"Synthetic test data is good enough on its own." → It's great for coverage but has errors. Anchor on curated real questions and spot-check the synthetic ones.
"Higher faithfulness is always better." → Not if it came from abstaining more — always read faithfulness with coverage. Improving one metric by wrecking another is a regression.
"Eval is a local script I run sometimes." → Wire it into CI as a gate. If a change regresses below target, it shouldn't merge — that's how you keep quality from drifting.

Key Takeaways

Evaluation replaces vibes with measurement — it's how you develop a RAG system, not a finishing touch. Un-measured is unmanaged.
Four RAGAS metrics, split by component: context recall + precision grade the retriever; faithfulness + answer-relevancy grade the generator. Targets: faithfulness > 0.9, answer-relevancy > 0.85, context-precision > 0.8, context-recall > 0.8.
A golden set is your ground truth — curate 20–50 real Q&A (cover the hard + unanswerable cases), expand with RAGAS synthetic generation, commit it, and grow it.
Evaluate your actual pipeline: for each golden question, run your retrieve() + generate(), then score with RAGAS (judge = Claude).
A low metric localizes the bug: low recall/precision → fix the retriever (L2); low faithfulness/relevancy → fix the generator (L3). Read retrieval first — generation can't beat its input.
Calibrate the judge against ~30–50 human labels (Cohen's κ) before trusting it — LLM judges have verbosity/position/self-enhancement bias.
Eval-driven development + CI gating: A/B contextual retrieval, choose min_score by plotting faithfulness vs. coverage, and gate every PR on the scores so regressions can't merge.
Next — L5 (Deploy & Observe): ship the assistant as a service on FastAPI + Modal and wire observability (Phoenix) so you can trace and debug every retrieve → rerank → generate in production — the last step of Project 1.