Evaluating RAG (RAGAS: Faithfulness, Relevance, Context Precision/Recall)

Introduction

Throughout this container, one phrase kept recurring: "measure it," "name the failure," "let each stage earn its place on your eval." This lesson is that eval — the measurement backbone that turns RAG from vibes-based tinkering into engineering. Because you have built a lot of machinery (chunking, hybrid, fusion, reranking, transforms, routing, graphs, multimodal, agentic), and the only way to know if any of it actually helped is to measure.

But here's the subtlety that makes RAG eval its own discipline: a single "is the answer good?" score is useless for fixing anything. RAG has two components that fail in completely different ways — bad retrieval and bad generation — and a blended score can't tell you which. The whole art is to decompose quality so you can diagnose and fix the right component.

In this lesson:

Why you must evaluate retrieval and generation separately
The four RAGAS metrics — and the RAG triad they map to
How RAGAS computes them with an LLM-as-judge
Diagnosing failures from the metric pattern — and the honest limits of LLM judges

Why a Single Score Isn't Enough

Imagine your RAG bot gives a wrong answer. Why? There are two fundamentally different culprits:

Retrieval failed — it never fetched the document containing the answer (or buried it under noise). The generator did its best with bad inputs.
Generation failed — retrieval surfaced the perfect context, but the model ignored it, hallucinated, or answered a different question.

These need opposite fixes — better embeddings/chunking/reranking vs. a better prompt/grounding/model — and a single "answer quality: 6/10" score tells you nothing about which. You'd be guessing. So RAG evaluation does what good debugging always does: it isolates the components. Measure retrieval on its own, measure generation on its own, and the pattern of scores points straight at the broken part. That decomposition is the entire value of frameworks like RAGAS (Retrieval-Augmented Generation Assessment).

An infographic titled 'Evaluating RAG — Measure Both Halves to Fix the Right One'. A strip: a single 'is the answer good?' score hides where it broke; RAG has two components, and the four RAGAS metrics measure each separately so you fix the right one (the RAG triad: Question-Context, Context-Answer, Question-Answer). Two groups. RETRIEVAL — did we fetch the right context: Context Recall (Context vs truth) — did we fetch ALL the needed context? under 100 percent means we missed something; Context Precision (Context vs Question) — are the relevant chunks ranked at the top, not buried in noise. GENERATION — did we use it well: Faithfulness (Answer vs Context) — is every claim grounded in the context? equals supported claims divided by total, catching hallucination; Answer Relevancy (Answer vs Question) — does the answer actually address the question, no off-topic or padding. A diagnosis card: the metric pattern diagnoses the fix — low recall means fix retrieval (chunking, embeddings, hybrid); low precision means rerank; low faithfulness means grounding prompt plus citations or Self-RAG; low relevancy means fix the prompt or query; diagnose recall first because it's the ceiling. A how-RAGAS-computes card: LLM-as-judge decomposes the answer into claims and NLI-checks each against the context (faithfulness), and generates questions from the answer (relevancy); mostly reference-free; auto-generate a synthetic test set. A caveat: LLM judges are noisy proxies — biased by length, position, and self-preference, only about 0.55 correlation with humans, and they don't verify factual accuracy (right entities plus a wrong answer can still score high); validate on a human-labeled sample, review edge cases, and don't chase one number. Banner: don't ask 'is it good?' — ask 'is retrieval or generation failing?'; decompose quality, diagnose the component, fix the right half.

The Four Metrics (and the RAG Triad)

RAGAS gives you four metrics, splitting cleanly into the retrieval pair and the generation pair. They map onto the RAG triad — the three relationships between the Question, the retrieved Context, and the Answer:

🔍 Retrieval metrics — did we fetch the right context?

Context Recall (Context ↔ ground truth) — Did we fetch all the information needed to answer? It's the fraction of the ground-truth answer's facts that are present in the retrieved chunks. Below 100% means retrieval missed something. (This is the one metric that needs a ground-truth answer.)
Context Precision (Context ↔ Question) — Is the relevant context ranked at the top, not drowned in noise? High precision = a clean signal; low = lots of irrelevant chunks diluting the good ones.

✍️ Generation metrics — did we use the context well?

Faithfulness (Answer ↔ Context) — Is every claim in the answer supported by the context? = supported claims ÷ total claims. This is the hallucination detector (the same grounding idea as L88's citations and L106's Self-RAG [ISSUP]).
Answer Relevancy (Answer ↔ Question) — Does the answer actually address the question — not off-topic, not padded with filler?

Together: Recall + Precision tell you about the librarian (retrieval); Faithfulness + Relevancy tell you about the writer (generation). A wrong final answer is one of their faults — and now you can tell whose.

How RAGAS Computes It: LLM-as-Judge

These aren't simple string comparisons — they're computed by an LLM acting as a judge, which is what makes them work on free-form text without exact-match reference answers. The cleverest is faithfulness:

# How FAITHFULNESS is computed — an LLM-as-judge pipeline, NOT a string match:
# 1) An LLM decomposes the answer into atomic CLAIMS.
#    Answer: "Password resets expire after 24h and require a verified email."
#      → ["resets expire after 24h", "resets require a verified email"]
# 2) An LLM runs an NLI check: is each claim SUPPORTED by the retrieved context?
#      claim 1 → in the docs ✅   |   claim 2 → NOT in the docs ❌  (a hallucination)
# 3) faithfulness = supported / total = 1/2 = 0.50

# Answer relevancy works the other way: an LLM generates questions FROM the answer,
# then measures cosine similarity of those to the ORIGINAL question (off-topic → low).

The mechanism is claim decomposition + NLI (natural language inference): the judge LLM breaks the answer into atomic claims, then checks each claim against the retrieved context — faithfulness is the fraction that are supported. This gives high-resolution feedback: you see exactly which sentence hallucinated. Answer relevancy runs it backwards — generate questions from the answer, then measure how close they are to the original question.

Crucially, three of the four metrics are reference-free — they need no labeled "correct answer" (only context recall does), which is what makes RAGAS practical at scale. And you don't need to hand-write the eval set: RAGAS's TestsetGenerator can synthesize a test set from your own documents. (For robustness, RAGAS even runs two independently-phrased judge prompts per metric and averages them, to resist prompt sensitivity.)

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

# An eval set mirroring your real query distribution (common, hard/multi-doc, edge cases).
data = Dataset.from_dict({
    "question":      [...],   # the user query
    "answer":        [...],   # YOUR system's generated answer
    "contexts":      [...],   # the chunks YOUR retriever returned (list per question)
    "ground_truth":  [...],   # the ideal answer (needed only for context_recall)
})

result = evaluate(data, metrics=[
    context_recall,      # RETRIEVAL: did we fetch everything needed?  (needs ground_truth)
    context_precision,   # RETRIEVAL: is the relevant context ranked at the top?
    faithfulness,        # GENERATION: is the answer grounded in the context?  (hallucination)
    answer_relevancy,    # GENERATION: does it address the question?
])
print(result)  # {'context_recall': 0.74, 'context_precision': 0.91, 'faithfulness': 0.62, ...}
# Change ONE thing → re-run → keep it only if the right metric moved (eval-driven dev, L102).

Diagnosis: Read the Pattern, Fix the Component

Here's the payoff — and it ties this entire container together. The pattern of which metrics are low tells you exactly which technique to reach for. Work it top-down, because recall is the ceiling:

If this metric is low…	The failure is…	Fix it with…
Context Recall	retrieval missed the doc	chunking (L90–92), hybrid search (L96), better embeddings, bigger k
Context Precision	retrieval is noisy / mis-ranked	reranking (L98), tune k
Faithfulness	the model hallucinated despite good context	grounding prompt + citations (L88), lower temp, Self-RAG (L106)
Answer Relevancy	answer is off-topic / incomplete	fix the prompt / query rewriting (L99), strip padding

Notice the discipline: you never fix retrieval when the problem is generation, or vice-versa. Try this in the widget below — flip metrics on and off and watch the diagnosis (and the prescribed fix) change:

Toggle each of the four RAGAS metrics ✅/❌ (or pick a failure scenario) and read the diagnosis — which component broke (retrieval vs generation) and the exact fix, mapped to the techniques from this whole container.

The single most important rule: diagnose Context Recall first. If recall is low, the right context was never retrieved — and no reranker, no prompt, no fancier model can fix it (the recall ceiling from L98). Fix recall, then precision, then faithfulness, then relevancy. Improving in the wrong order is wasted effort.

The Honest Take: Trust, but Verify the Judge

LLM-as-judge eval is scalable and genuinely useful — but it is not ground truth, and treating it as gospel will burn you. Be clear-eyed:

LLM judges are biased. They favor longer answers (verbosity bias), show position bias, and prefer outputs from their own model family (self-enhancement) — a dozen such biases are documented. A higher score isn't always a better answer.
They're noisy proxies. Correlation between automated RAG metrics and human judgment runs around ~0.55 (and ~85–92% agreement at best). Good enough to catch big regressions and rank versions; not good enough to trust a 0.02 difference.
They don't verify factual correctness. This is the big one: a passage with the right entities but the wrong answer scores high on relevance across every tool. These metrics check grounding and relevance, not truth — a confidently-wrong-but-well-grounded answer can sail through.
Garbage references = garbage scores. Context precision/recall need good ground-truth contexts; without them some tools silently degrade to "the LLM guesses what the context should have been."

So use eval the right way: (1) Validate the judge against a small human-labeled sample before you trust its numbers. (2) Budget human review on edge cases — experts catch the subtle errors metrics miss; humans remain the gold standard. (3) Don't optimize a single number — the value is the decomposition (which component to fix), not a leaderboard score. (4) Run the eval in CI (eval-driven development, L102) so every change to your stack is gated on the metric it's supposed to move. Other frameworks — TruLens (the RAG triad), DeepEval, Arize Phoenix — do the same job; RAGAS is the canonical RAG-specific one.

🧪 Try It Yourself

Diagnose with the widget, then reason about real cases:

In the interactive, set only Faithfulness to ❌ (the rest ✅). What's the diagnosis, and why is it a generation problem, not a retrieval one?
Now set Context Recall ❌ with everything else ✅. Why must you fix this before touching the reranker or the prompt?
An answer about "password resets expire after 24h and require a verified email" is graded faithfulness = 0.5. What does that 0.5 literally mean?
Your nightly eval shows answer relevancy dropped after a prompt change, but faithfulness and context scores are unchanged. What broke, and where do you look?
Your faithfulness scores are all 0.95+, yet users report wrong answers. How is that possible — and what do you add to catch it?

→ (1) Hallucination — the context was good (retrieval metrics pass) but the answer made unsupported claims; fix generation (grounding prompt + citations, lower temp, Self-RAG), not retrieval. (2) Low recall means the answer's context was never retrieved — it's the ceiling; a reranker can only reorder what's there, and a prompt can't ground in absent context. Fix retrieval (hybrid/chunking/k) first. (3) The answer made 2 claims; 1 is supported by the context and 1 isn't (a hallucination) → 1/2 = 0.5. (4) Answer relevancy fell → the new prompt made answers off-topic / padded / incomplete; look at the prompt (and query understanding), not retrieval. (5) Faithfulness checks grounding, not truth — the answer can be faithfully grounded in wrong or outdated context. Add a factual-correctness check (vs a ground-truth answer) and human review of edge cases.

Mental-Model Corrections

"One 'answer quality' score is enough." It hides which component failed. Decompose into retrieval (recall, precision) and generation (faithfulness, relevancy) to know what to fix.
"Faithfulness = correctness." No — faithfulness checks if the answer is grounded in the retrieved context, not whether that context (or answer) is true. A well-grounded answer from wrong context scores high.
"Context Precision and Recall are the same." Recall = did we fetch all needed context (the ceiling)? Precision = is the relevant context ranked well / not noisy? Low recall → improve retrieval; low precision → rerank.
"RAGAS needs labeled answers." Mostly reference-free (only context recall needs ground truth); it can even synthesize a test set from your docs.
"LLM-judge scores are objective truth." They're biased, noisy proxies (~0.55 human correlation) that don't verify facts. Validate against humans; review edge cases.
"Chase the highest average score." The value is the diagnostic decomposition (fix the right component), run in CI — not a single leaderboard number.

Key Takeaways

Evaluate retrieval and generation separately — a single score can't tell you which half failed, and they need opposite fixes.
Four RAGAS metrics / the RAG triad: Context Recall (fetched all needed context? — the ceiling) + Context Precision (relevant context ranked well?) for retrieval; Faithfulness (answer grounded in context? = supported claims ÷ total → hallucination) + Answer Relevancy (addresses the question?) for generation.
Computed by LLM-as-judge: decompose the answer into claims and NLI-check each vs context (faithfulness); generate questions from the answer (relevancy). Mostly reference-free; auto-synthesize a test set.
Diagnose by pattern (recall first): low recall → retrieval (chunk/embed/hybrid); low precision → rerank; low faithfulness → grounding/Self-RAG; low relevancy → prompt/query.
Honest limits: LLM judges are biased, noisy proxies (~0.55 human correlation) that don't verify truth (right entities + wrong answer can score high). Validate against humans, review edge cases, run in CI, and use the decomposition — not one number.
Next, the finale: the RAG Design Decision Guide — how to choose, from everything in this container, the right stack for your problem.