Evaluation vs Baseline

Introduction

In L12 (LoRA Fine-Tuning Run) your loss curves flattened and you saved an adapter. Converging proved training worked — it did not prove the model is good. A flat loss curve can hide over-memorization, and it says nothing about whether your small tuned student actually beats the Claude baseline at triage. This lesson is where you find out — rigorously — and make the call: ship it, or don't.

Here's the principle that governs everything: a fine-tune's value is the delta vs a baseline on the same test set. A score of "91% accuracy" in isolation is meaningless; "91%, up from the base model's 62% and matching the Claude baseline's 92% — at 1/20th the cost" is a decision. So you'll score the tuned model and its baselines on the sacred held-out test set you carved in L11 (the one that never touched training or synthesis) and read the difference.

You'll learn to:

Measure the task — accuracy, per-class (a confusion matrix, because aggregate accuracy hides minority-class failures), and schema-valid JSON %.
Compare to the right baselines — the untuned base (did tuning do anything?) and the prompted frontier model (is the fine-tune even worth it?).
Prove it's real — a bootstrap confidence interval on the delta, so you ship on the lower bound, not the mean.
Catch what fine-tuning breaks — catastrophic forgetting (general capability), and weigh the cost / latency win that justified the whole project.

Why this is a senior skill. Anyone can report a number that looks good. Knowing whether a lift is real, un-contaminated, and worth shipping — and being honest when the answer is no — is what makes you trustworthy with a model. We use the course stack: scikit-learn for metrics, Claude claude-sonnet-4-6 as the baseline and (where needed) the judge.

Scope: this lesson owns proving the model is better. Deploying it → L14 (Serving the Model).

Hero infographic titled 'Evaluation vs Baseline' for the third lesson of the fine-tune-and-ship project, on a white background. The deck says: converging proved training worked — now prove the tuned model is actually better; the fine-tune's value is the DELTA vs the baseline on the sacred held-out test set. The centre is a head-to-head comparison: a small fine-tuned student model and the Claude baseline both run the same untouched test set (drawn with arrow connectors), feeding three score bars — task accuracy (the delta vs base), win-rate head-to-head, and general capability (a catastrophic-forgetting check) — and a bootstrap confidence interval whisker on the delta with a dashed zero line, captioned 'ship on the LOWER bound, not the mean'. A panel of FOUR SHIP GATES is shown: no contamination, statistically significant (CI lower bound above zero), no catastrophic forgetting, and a lift big enough to justify fine-tuning. A cost-and-latency callout shows the small tuned model is about 20 times cheaper and faster than the frontier baseline. Three summary cards along the bottom read: 'The value is the delta vs the base on the same clean test'; 'Is it real? bootstrap the CI, ship on the lower bound'; 'Did it forget? check general capability, not just the task'. A family strip lists the four project lessons — Brief & Data Curation, LoRA Fine-Tuning Run, Evaluation vs Baseline, Serving the Model — with the third highlighted. Slate, sky, cyan, violet, amber, emerald accents; generous legible text.

The Value Is the Delta — Baselines to Beat

A fine-tuned model's quality is never an absolute number — it's a comparison. To know if your tune is worth anything, score it against three references, all on the same sacred test set, same week, same rubric:

The untuned base model — your floor. If the tune barely beats the raw base, training didn't do much. This delta proves the fine-tune learned something.
The prompted frontier model (Claude) — your quality bar. This is the baseline you were trying to match cheaply. If a well-crafted prompt on the base model (or Claude) already matches your tune, the fine-tune may not be justified — a prompt costs nothing to maintain.
The previous version (once you have one) — your regression gate. A new tune must not go backwards on cases the old one passed.

Score the base on the same contract — same examples, same rubric, same judge, same week — and the fine-tune's value is the delta. This is why the sacred held-out test set from L11 matters so much: if it leaked into training (or into the synthetic generator that saw it), your delta is inflated by memorization, not skill — a contaminated benchmark makes a bad model look great. Everything in this lesson rests on that test set being clean.

Measuring the Task — Accuracy, Per-Class, Schema-Valid

For our triage model, three task metrics matter — and the second is where beginners get fooled:

Accuracy — the fraction of tickets where the predicted category (and priority) matches the gold label. Simple, but it lies on imbalanced data. If 80% of tickets are billing, a model that always says billing scores 80% accuracy while being useless on everything else.
Per-class — the confusion matrix. Break accuracy down per category (precision/recall/F1 for each), and use the macro-F1 (the unweighted average across classes) so minority classes count equally. This is what reveals "it's great at billing and blind to outages." The confusion matrix shows exactly which categories get mistaken for which — your error-analysis map.
Schema-valid JSON % — the fraction of outputs that parse as JSON in the taxonomy. This is the format win fine-tuning was for: a good tune should hit ~100%, where a prompted model drifts. If schema-validity is low, nothing else matters — a broken output can't be triaged.

Compute them with scikit-learn: an accuracy_score, a classification_report (per-class precision/recall/F1), and f1_score(average="macro"). Always read the per-class breakdown — the aggregate number is where bad models hide.

Running the Eval — Score on the Sacred Test Set

Load the base model + your LoRA adapter (PEFT stacks them), run every example in the untouched test.jsonl, parse the output, and score it against the gold labels:

# eval/run.py — score the tuned model on the SACRED held-out test set
import json
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
from sklearn.metrics import accuracy_score, f1_score, classification_report

base  = AutoModelForCausalLM.from_pretrained(BASE, device_map="auto")
tuned = PeftModel.from_pretrained(base, "out/triage-lora")     # base + LoRA adapter
tok   = AutoTokenizer.from_pretrained(BASE)

test = [json.loads(l) for l in open("data/test.jsonl")]        # NEVER seen in training
y_true, y_pred, schema_ok = [], [], 0
for ex in test:
    gold = json.loads(ex["messages"][-1]["content"])
    out  = generate_json(tuned, tok, ex["messages"][:-1])       # the model's triage
    try:
        pred = json.loads(out); schema_ok += 1                   # valid JSON?
    except ValueError:
        pred = {"category": "INVALID"}                          # a broken output is a miss
    y_true.append(gold["category"]); y_pred.append(pred.get("category"))

print("accuracy   :", accuracy_score(y_true, y_pred))
print("macro-F1   :", f1_score(y_true, y_pred, average="macro"))   # minority classes count
print("schema-valid:", schema_ok / len(test))
print(classification_report(y_true, y_pred))                       # per-class confusion

Run the exact same loop for your baselines — the untuned base model and the prompted Claude baseline — so all three numbers come from the same examples and the same scorer. Now you have a scorecard: tuned vs base vs Claude on accuracy, macro-F1, and schema-validity. But a scorecard isn't a decision yet — you have to know whether the difference is real.

Is It Real? — Statistical Significance & the Bootstrap CI

Your tuned model scores +9 points over the base. Ship it? Not yet. On a small test set, a 9-point gap can be pure noise — flip a few examples and it vanishes. The professional move is to put a confidence interval on the delta and ship on its lower bound, not the mean.

The bootstrap is the simplest honest way to get that CI: take the per-example win/loss difference between the two models, resample it with replacement a thousand times, and read the 2.5th and 97.5th percentiles. If the lower bound is above 0, the lift is real; if the interval crosses 0, you cannot rule out "no improvement."

# eval/bootstrap.py — is the lift real, or judge noise?
import numpy as np

def delta_ci(tuned_correct, base_correct, n_boot=1000):
    """tuned_correct / base_correct: per-example 0/1 arrays on the SAME test set."""
    diff = np.array(tuned_correct) - np.array(base_correct)        # per-example difference
    boots = [np.mean(np.random.choice(diff, len(diff), replace=True))
             for _ in range(n_boot)]                               # resample with replacement
    lo, hi = np.percentile(boots, [2.5, 97.5])
    return diff.mean(), lo, hi                                     # SHIP only if lo > 0

mean, lo, hi = delta_ci(tuned_correct, base_correct)
print(f"delta = {mean:+.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]  ->",
      "REAL" if lo > 0 else "NOISE — need a bigger eval set or a larger lift")

The rule of thumb the research is blunt about: "a 0.3-point lift on 50 examples is judge noise." The fix is more eval data (a wider, cleaner test set shrinks the CI) or a bigger true lift. This single discipline — ship on the lower bound — separates teams that ship real improvements from teams that ship random walks and wonder why production didn't move.

Did It Forget? — Catastrophic Forgetting

Here's the silent regression that catches everyone: you fine-tune for triage, the task score goes up, you ship — and the model is now dumber at everything else. That's catastrophic forgetting: by adapting to your narrow task, the model overwrites general capability it used to have.

So a complete evaluation measures two things, not one:

Task performance — up (the lift you wanted).
General capability — held. Run the tuned model on a small general benchmark (a slice of MMLU, GSM8K, or your own general-instruction set) and compare to the base. If it dropped, you traded breadth for depth.

The research nuance worth knowing: fine-tuning tends to degrade instruction-following more than factual knowledge — the model still knows things, but gets worse at following general instructions outside your task. Two things protect you here: you already used LoRA (a frozen base forgets far less than full fine-tuning — a payoff from L12), and the fixes when it does forget are familiar — fewer epochs, a lower learning rate, more diverse data, or general-data replay. For a narrow first-pass triage model some forgetting may be acceptable; for a model that must stay a generalist, a big general-capability drop is a don't-ship.

The Real Win — Cost & Latency

Step back to the brief: we didn't fine-tune to beat Claude on quality — we fine-tuned to match it cheaply. So the evaluation isn't complete until you've measured the thing the whole project was for: cost and latency.

Cost per 1k tickets — a small open model (3–8B) served on your own hardware is often ~10–20× cheaper per call than a frontier API. At thousands of tickets a day, that's the difference between a viable feature and a line item that gets cut.
Latency — a frontier model might take 200–300 ms per classification; a small local model responds in tens of ms. For a high-volume first-pass triage, that speed is the product.

Now the trade-off is explicit and honest: "the tuned 3B matches the Claude baseline within the CI on task accuracy and holds general capability, at ~20× lower cost and ~5× lower latency." That's a ship. The opposite is just as important to be able to say: if the tuned model is 3 points worse but 20× cheaper, is that worth it? It depends on the cost of a wrong triage — and naming that trade-off out loud, with numbers, is the senior move.

The Ship Decision — Four Gates

Pulling it together: a fine-tune ships only if it clears all four gates. Fail any one and the answer is don't ship (or fix and re-measure):

🧪 Not contaminated. The test set is clean — no overlap, near-duplicates, or leakage through a synthetic generator that saw it (L11's decontamination). A contaminated score is a lie; check this first, because nothing below means anything if it fails.
📊 Statistically real. The bootstrap CI lower bound on the delta is > 0. Ship on the lower bound, not the mean — a small lift on a small eval set is noise.
🧠 No catastrophic forgetting. General capability held (within an acceptable drop). The model got sharper on the task without getting dumber elsewhere.
💰 The lift earns its keep. The improvement (and/or the cost/latency win) is big enough to justify maintaining a fine-tune over a good prompt. A marginal lift over a well-prompted base isn't worth the operational cost.

And one cross-cut: add human spot-checks on 5–10% of cases to confirm the metric agrees with human judgment. The interactive below lets you trip each gate in turn — turn on contamination, shrink the eval set, crank the forgetting — and watch the verdict flip. A fine-tune is only as good as your proof that it's better.

Wiring It All Together — eval.py

The whole evaluation, assembled — score all three models on the sacred test, bootstrap the delta, check forgetting and cost, and emit a ship / don't-ship verdict:

# eval.py — the full ship gate for the fine-tune
from eval.run import score_model            # accuracy, macro_f1, schema_valid, per_example_correct
from eval.bootstrap import delta_ci
from eval.general import general_capability  # small MMLU/GSM8K slice -> score
from eval.cost import cost_latency           # $/1k + ms/call per model

test = load_jsonl("data/test.jsonl")        # SACRED — never trained on

tuned  = score_model("out/triage-lora", test)
base   = score_model(BASE, test)            # untuned floor
claude = score_model("claude-sonnet-4-6", test)  # the prompted baseline / quality bar

mean, lo, hi = delta_ci(tuned.per_example_correct, base.per_example_correct)
forgot = general_capability(BASE) - general_capability("out/triage-lora")   # pp dropped
cl, bl = cost_latency("out/triage-lora"), cost_latency("claude-sonnet-4-6")

ship = (not test.contaminated            # gate 1: clean test
        and lo > 0                        # gate 2: CI lower bound > 0
        and forgot <= 5                   # gate 3: general capability held
        and tuned.accuracy >= claude.accuracy - 0.02)   # gate 4: matches the bar...
print("SHIP" if ship else "DON'T SHIP",
      f"| task {tuned.accuracy:.2f} (Δ {mean:+.2f}, CI≥{lo:+.2f}) | schema {tuned.schema_valid:.2f}"
      f" | forgot {forgot:.0f}pp | {bl.cost/cl.cost:.0f}x cheaper, {bl.ms/cl.ms:.0f}x faster")

That's the honest verdict, in one line: does the tuned student match the baseline within the CI, stay a generalist, and win on cost — on a clean test set? If yes, you've earned the right to ship it — which is L14 (Serving the Model). If no, the eval just told you exactly what to fix (more data, fewer epochs, decontaminate) before you waste a deploy.

✅ Definition of Done (this step)

Before L14, your model should be proven, not assumed:

Scored on the sacred held-out test set (L11) — confirmed clean (no leakage/contamination).
Task metrics reported — accuracy, macro-F1, per-class confusion, and schema-valid JSON % — and read the per-class breakdown.
Compared against both baselines — the untuned base (the tune learned something) and the prompted Claude (the tune is worth it) — on the same examples.
The lift is statistically real — a bootstrap CI on the delta with lower bound > 0 (you ship on the bound, not the mean).
General capability checked — no unacceptable catastrophic forgetting.
The cost/latency win quantified, and a clear ship / don't-ship decision (all four gates) — plus human spot-checks on a sample.

If you can state, with numbers and a CI, "it matches the baseline, holds general ability, and is 20× cheaper — on a clean test" — ship it in L14. If not, you know exactly what to fix.

See It — The Fine-Tune Eval Lab

This lab turns the ship decision into something you can feel. Drag the four knobs — task lift, eval-set size N, general-capability drop, win-rate — and toggle the gotchas (contamination, position-bias correction, human spot-checks). The three dimension cards and the bootstrap CI update live, and a ship / don't-ship verdict fires on the four gates.

Set a healthy +9pp lift but a tiny eval set → watch the CI widen past 0: "not statistically real — judge noise." Now grow N → the CI tightens above 0 → real.
Flip contamination ON → the task score inflates and the verdict screams "the score is a lie."
Crank the general-capability drop → catastrophic forgetting blocks the ship even with a great task lift.
Drop the lift below ~3pp → "a good prompt probably matches it" — fine-tuning isn't justified.

The Fine-Tune Eval Lab — evaluate the tuned model against its baseline, then make the ship call, live. Drag the four knobs — task lift vs base, eval-set size N, general-capability drop, and win-rate — and toggle the gotchas (train/eval contamination, position-bias correction, human spot-checks). The lab scores the fine-tune across the three dimensions that gate shipping (task performance vs base, head-to-head win-rate, and general-capability regression for catastrophic forgetting) and drives a live bootstrap 95% confidence interval on the delta — shrink the eval set and watch the CI widen until it crosses zero ('a +9pp lift on 60 examples is judge noise'). A SHIP / DON'T-SHIP verdict fires on the four real gates: contamination (the score is a lie), statistical insignificance (ship on the lower bound, not the mean), catastrophic forgetting (sharper on the task, dumber on everything else), and a lift too small to justify the fine-tune over a good prompt. The lesson lands in your hands: a fine-tune is only as good as your proof that it's better. Deterministic teaching model, not a live eval run.

Notice the discipline. The mean delta is not the decision — the CI lower bound is. A clean test, a significant lift, no forgetting, and a lift worth the cost: only when all four are green do you ship. A fine-tune is only as good as your proof that it's better.

🧪 Try It Yourself

Reason these out, then check against the lab and the code.

1. Your tuned model scores 94% accuracy on triage. Is that good? What's the one extra number you need before you can say?

2. The tune beats the base by +6pp on a 40-example test set. Marketing wants to ship. What do you measure first, and what might it say?

3. Accuracy is a strong 88%, but the model never correctly labels outage tickets (the rarest 5%). Which metric exposes this, and why did accuracy hide it?

4. Your tune is +10pp on triage but −12pp on a general-reasoning benchmark. Ship? What's the name for this, and what reduces it?

5. The tuned 3B is 2pp worse than the Claude baseline on accuracy but 20× cheaper and 5× faster. Ship or not?

Answers.

1. You can't tell yet — an absolute score is meaningless. You need the delta vs a baseline on the same test set: 94% is great if the base scored 60% and Claude scores 95%, and useless if a plain prompt on the base already hits 93%. The value is the delta.

2. Measure statistical significance — a bootstrap CI on the delta. On only 40 examples, a +6pp lift likely has a CI that crosses 0, meaning it could be pure noise. Ship on the lower bound, not the mean — get a bigger, clean eval set before claiming a win.

3. The per-class confusion matrix / macro-F1 exposes it — outage has 0% recall. Accuracy hid it because outage is only 5% of the data, so getting it always-wrong barely dents the aggregate. Always read the per-class breakdown on imbalanced tasks.

4. Don't ship (for a generalist) — this is catastrophic forgetting: the model got sharper on the task and dumber on everything else. Reduce it with fewer epochs, a lower learning rate, more diverse data / general-data replay, or by relying on LoRA's frozen base (which forgets far less than full fine-tuning). For a narrow first-pass model some drop may be acceptable — name the trade-off.

5. It depends — and that's the honest answer. The whole point was to match quality cheaply, so if a triage error is low-cost (a human reviews escalations anyway), 2pp for 20× cheaper + 5× faster is an easy ship. If a wrong triage is expensive (a P1 outage mislabeled as billing), the 2pp may not be worth it. Put the cost of the error against the cost saving — with numbers — and decide.

Mental-Model Corrections

"Converging means the model is good." → Converging means training worked. Good means it beats the baseline on a clean held-out test — a different, harder claim.
"94% accuracy is a great score." → A score in isolation is meaningless. The value is the delta vs a baseline on the same test — vs the untuned base and a well-prompted model.
"A higher mean means it's better." → Not if the CI crosses 0. A small lift on a small eval set is noise — ship on the bootstrap lower bound, not the mean.
"Accuracy tells me how good it is." → On imbalanced data, accuracy hides minority-class failure. Read the per-class confusion matrix and macro-F1.
"Just measure the task." → Also measure general capability — fine-tuning can cause catastrophic forgetting (sharper on the task, dumber elsewhere). Measure both.
"If it leaked a little, the score's still roughly right." → A contaminated test inflates the score by memorization, not skill — the number is a lie. Decontaminate and re-measure; check this first.
"A fine-tune that beats the base should ship." → Only if it beats a well-prompted baseline by enough to justify the ops cost. A marginal lift over a good prompt isn't worth maintaining a model.
"Quality is the only thing that matters." → The brief was to match quality cheaply. Cost and latency are first-class results — a 2pp gap can be an easy ship at 20× cheaper, depending on the cost of an error.

Key Takeaways

A fine-tune's value is the delta vs a baseline on the same sacred held-out test set — vs the untuned base (did it learn?) and the prompted frontier model (is it worth it?). A score in isolation is meaningless.
Measure the task properly: accuracy and the per-class confusion matrix / macro-F1 (aggregate accuracy hides minority-class failure), plus schema-valid JSON % (the format win).
Prove it's real: a bootstrap CI on the delta — ship on the lower bound, not the mean. A small lift on a small eval set is noise; a contaminated test is a lie (check it first).
Check what tuning breaks: catastrophic forgetting — measure general capability, not just the task. LoRA's frozen base helps; fewer epochs / lower LR / diverse data fix it.
Weigh the real win: cost & latency. The point was to match quality cheaply — a small tuned model at ~20× lower cost and ~5× lower latency can be a ship even a couple points behind, depending on the cost of an error.
The ship decision = four gates: clean test · statistically significant · no forgetting · lift worth the cost (+ human spot-checks). A fine-tune is only as good as your proof that it's better.
Next — L14 (Serving the Model): with a proven, shippable model, merge or load the adapter and serve it — quantize, deploy, and put it behind an endpoint so it actually triages tickets in production.