Evaluating a Fine-Tuned Model

Introduction

You've learned to train models (SFT, DPO, GRPO) and to combine them (merging). This lesson answers the question that decides whether any of it was worth doing: did the fine-tune actually make the model better — and can you prove it?

This is the discipline that separates a demo from a shipped model. The governing idea: a fine-tune is only as good as your ability to measure it. And the single most important rule — the one beginners miss — is that you never report a fine-tune's absolute score. You report the delta vs the base model, measured on the exact same held-out test. If the base + a good prompt does just as well, the fine-tune wasn't justified. By the end you'll evaluate across three dimensions, prove a result is real (not judge noise), and make a clean ship / don't-ship call.

An infographic titled 'Evaluating a Fine-Tuned Model', the fifth lesson of section four on advanced post-training, on how to prove a fine-tune actually made the model better. The governing principle is that a fine-tune is only as good as your ability to measure it, and its value is the delta, not the absolute score: you score the base model and the candidate on the very same held-out test, with the same examples, the same rubric, and the same judge, and the improvement of the candidate over the base is the only thing that justifies fine-tuning. The eval harness must exist before training, with pass-fail criteria defined upfront and a test set isolated from the training data so there is no leakage. Evaluation spans three dimensions. First, task performance: a task-specific metric for classification or extraction, or an LLM-as-judge or human rating for open-ended generation, measured as the delta over the base. Second, win-rate: a pairwise judge compares the candidate against a baseline and reports win plus half of ties over the total, as in AlpacaEval and MT-Bench, run in both orders to cancel the judge's position bias. Third, general-capability regression: the silent killer called catastrophic forgetting, where the model gets sharper on the narrow task but dumber on everything else, detected by scoring a general benchmark such as MMLU or GSM8K on the base and the candidate. Two further disciplines guard the verdict. Statistical rigor: a small lift on a small evaluation set is judge noise, so bootstrap a confidence interval on the per-example delta by resampling, and ship on the lower bound rather than the mean. And contamination control: if the test overlaps the training data, the score is inflated by memorization and is a lie. The takeaway is to measure the delta against the base across task, win-rate, and general capability, confirm it is statistically real and uncontaminated, and only then ship.

The One Rule: Measure the Delta vs the Base

Everything in fine-tuning evaluation hangs on one principle: the fine-tune's value is the delta.

Score the base model and your candidate on the same contract — same held-out examples, same rubric, same judge — and the candidate's improvement over the base is the only number that matters. An absolute "87% accuracy" is meaningless on its own: maybe the base already scored 86% with a decent prompt, and you spent a week and a GPU to gain one point you could've gotten for free.

Two non-negotiables that make the delta trustworthy (both from L200 (Build a Fine-Tuning Dataset)):

The eval harness exists before training. Define your pass/fail criteria upfront — not after you see the numbers (or you'll rationalize whatever you got).
Strict train/eval isolation. The test set must have zero overlap with training data, or the score is inflated by memorization (the L199 contamination problem — you'll see it break the verdict in the lab).

With the delta as your north star, you evaluate it across three dimensions.

Dimension 1: Task Performance

First, did the model get better at the thing you fine-tuned it for? Measured on the held-out test, vs the base:

Closed tasks (classification, extraction, structured output) → exact-match / accuracy / F1 — cheap, deterministic, unambiguous.
Open-ended generation (summaries, chat, code-with-explanation) → there's no single right string, so use an LLM-as-judge with a rubric (score each output against explicit criteria) at scale, plus human spot-checks on 5–10% to confirm the judge is calibrated.

Report it as base X% → candidate Y%, and keep the slices: overall is fine, but break it down by input category (the L195 coverage map) — a fine-tune often helps the common cases and hurts the edge ones. The number you ship on is the delta, sliced where it matters.

Dimension 2: Win-Rate (Pairwise vs a Baseline)

For generation, a raw judge score ("7.2/10") is shaky — humans and LLMs are far more reliable at comparing two answers than scoring one (the same insight that powers DPO, L202). So the standard for open-ended quality is the pairwise win-rate:

Put the candidate and a baseline (the base model, or a reference like GPT-class) head-to-head on each prompt; a judge picks win / lose / tie.
Win-rate = (wins + ½·ties) / total. Above 50% beats the baseline. This is exactly what AlpacaEval 2.0 and MT-Bench report.

The trap: position bias. LLM judges tend to favor whichever answer comes first — so you run both orders (candidate-first and baseline-first) and average. Skip that and your win-rate is an artifact of ordering, not quality (you'll see this toggle in the lab). Watch the disagreements too: if task accuracy went up but win-rate didn't, the metric and human preference are telling you different things.

Dimension 3: Regression — Catastrophic Forgetting

This is the dimension people forget to measure — and it's the #1 silent regression in fine-tuning. When you sharpen a model on a narrow task, it often gets dumber on everything the task didn't exercise. That's catastrophic forgetting: the fine-tune wins on your benchmark while quietly losing general knowledge, reasoning, instruction-following, and safety behavior.

You only catch it if you look for it. Score a general-capability benchmark — MMLU (knowledge), GSM8K/ARC (reasoning), instruction-following, safety/refusals — on both the base and the candidate. If those drop, you have forgetting, even if your task metric soared. Build these regression slices into the harness so you catch it before shipping.

Causes & fixes (all from earlier lessons): too-high LR or too many epochs (L194) → use fewer; full fine-tuning is more destructive than LoRA (L192); gradient conflict between the task and general data accelerates forgetting. Mitigations: replay some general data in the mix, lower the LR, prefer LoRA/PEFT, or merge the fine-tune back toward the base (L204). The rule: a task win that comes with a capability loss is usually not worth shipping.

Is It Real? Statistical Rigor & Contamination

A delta on a screen isn't proof. Two checks decide whether your result is real:

1. Statistical significance. A +0.3-point lift on 50 examples is judge noise, not a win. Quantify the uncertainty: bootstrap the confidence interval — resample the per-example differences with replacement ~1,000 times — and ship on the lower bound of the CI, not the mean. A small eval set gives a wide CI that crosses zero (you can't rule out "no improvement"); a bigger eval set or a larger true lift tightens it above zero. (This is the Container-4 statistical-rigor discipline applied to fine-tuning.)

2. Contamination. If your test set overlaps the training data (or a public benchmark the model has seen), the score is inflated by memorization — it's a lie, not a measurement (L199 decontamination). Always confirm the eval is clean before you trust a single number.

Together: measure the delta, bound its uncertainty, and verify the test is clean. Only a lift whose CI lower bound is above zero on uncontaminated data counts as a win. You'll feel all of this — shrink the eval set and watch the CI swallow zero — in the lab.

In Code: The Eval Harness

The harness scores base and candidate on the same test, computes the delta, bootstraps the CI, and checks the general-capability slice — the whole discipline in one place:

import numpy as np

# Score BASE and CANDIDATE on the SAME held-out, decontaminated test (same judge/rubric).
base_scores = evaluate(base_model, test_set)        # per-example: 1.0 / 0.0 (or judge score)
cand_scores = evaluate(candidate,  test_set)
delta = cand_scores.mean() - base_scores.mean()     # the fine-tune's VALUE is the delta

# BOOTSTRAP a 95% CI on the per-example difference - ship on the LOWER bound.
diff = cand_scores - base_scores
boot = [np.random.choice(diff, len(diff), replace=True).mean() for _ in range(1000)]
lo, hi = np.percentile(boot, [2.5, 97.5])
significant = lo > 0          # a +0.3 lift on 50 examples will NOT pass this

# REGRESSION slice - catch catastrophic forgetting on general capability.
gen_delta = evaluate(candidate, mmlu).mean() - evaluate(base_model, mmlu).mean()

ship = significant and gen_delta > -0.02      # real task lift AND no general-capability loss
# (For generation: also a pairwise win-rate vs baseline, judged in BOTH orders.)

See It: The Fine-Tune Eval Lab

Make the ship call yourself. Set a task lift, then shrink the eval set and watch the bootstrap CI widen until it swallows zero — your +5pp is suddenly judge noise. Drag up the general-capability drop to trigger catastrophic forgetting (don't ship!). Flip on contamination and watch the verdict call your score a lie. Turn off position-bias correction and see the win-rate become unreliable. Only when the delta is real, uncontaminated, and regression-free does it say SHIP.

Interactive: the Fine-Tune Eval Lab. The user evaluates a fine-tune against its base and decides whether to ship. Sliders set the task lift over the base, the evaluation set size, the drop in general capability, and the win-rate against a baseline; toggles model train-eval contamination, position-bias correction by running both orders, and human spot-checks. Three dimension cards show base versus fine-tune for task performance and general capability with the delta, and a win-rate gauge against the fifty-percent line. A bootstrap ninety-five-percent confidence interval on the task delta is drawn against a zero line, and the eval-set-size slider widens or narrows it, so a small lift on a small set produces a CI that crosses zero, meaning the gain could be pure noise; the rule shown is to ship on the lower bound, not the mean. A ship-or-don't verdict fires on the four real gates: contamination making the score a lie, statistical insignificance when the CI crosses zero, catastrophic forgetting when general capability drops too far, and a lift too small to justify fine-tuning at all. An arrow workflow runs base-plus-fine-tune to the same held-out test to the three dimensions and the bootstrap to the ship decision.

Four gates, one decision: real lift (CI lower bound > 0), clean test (no contamination), no forgetting (general capability held), and worth it (beats base + prompting). Miss any one and you don't ship.

Why This Matters

This is the lesson that keeps you honest — and keeps a fine-tune from quietly making your product worse. The history of fine-tuning is full of teams who celebrated a benchmark jump and shipped a model that had forgotten how to follow basic instructions, or whose "win" was judge noise on 40 examples, or whose eval was contaminated by the very data they trained on. Knowing how to measure the delta, bound it statistically, and check for regression is what lets you say — with evidence — "this fine-tune is better, ship it" or "this isn't worth it, the base does fine." That judgment is the whole point of post-training. Next, you put all of Section 4 (and the section before it) together: L206 (Hands-On: Fine-Tune a Small Model with LoRA) — train, and evaluate, a real model end to end.

🧪 Try It Yourself

Find all four ship-blockers in the Fine-Tune Eval Lab:

The noise mirage. Set a +5pp lift with eval set N = 40. Does the bootstrap CI lower bound clear zero? Now raise N until it does. Why is "+5pp" meaningless without N?
The silent loss. Set a big task lift but drag general-capability drop to −12pp. Why does the lab refuse to ship — and what is this failure called?
The contaminated win. Turn on contamination with a healthy lift. Why is the impressive task number now a lie?

Bonus: your fine-tune scores 87% and the base scores 85% with a good prompt. In one sentence, should you ship the fine-tune — and what's the real question you should be asking?

Mental-Model Corrections

"Report the fine-tune's accuracy." Report the delta vs the base on the same test — the absolute number alone tells you nothing about whether fine-tuning helped.
"The task metric went up, so ship it." Not until you check general-capability regression (catastrophic forgetting) and statistical significance. A task win can hide a capability loss.
"A higher score is a real improvement." A small lift on a small eval set is judge noise. Bootstrap the CI and ship on the lower bound.
"An LLM judge is objective." It has position bias (favors the first answer), verbosity bias, and self-preference. Run both orders, and spot-check with humans.
"Build the eval after training." Build it before — define pass/fail upfront and keep the test decontaminated, or you'll fool yourself.

Key Takeaways

A fine-tune is only as good as your proof. Always measure the delta vs the base on the same held-out, decontaminated test (same rubric, same judge).
Three dimensions: task performance (accuracy / LLM-as-judge), win-rate (pairwise vs baseline, both orders for position bias), and general-capability regression — catch catastrophic forgetting by scoring MMLU/GSM8K base-vs-candidate.
Prove it's real: a small lift on a small set is judge noise — bootstrap the CI and ship on the lower bound, not the mean.
Keep it clean: contamination inflates scores into a lie — decontaminate first (L199).
The harness exists before training, with pass/fail criteria and regression slices built in.

Next: L206 — Hands-On: Fine-Tune a Small Model with LoRA — the capstone that trains and evaluates a real model end to end.