Project Brief & Data Curation

Welcome to Project 3 — Fine-Tune & Ship a Specialized Model. In Projects 1 and 2 you built systems around a frontier model (a RAG assistant, then a multi-tool agent). Now you'll change the model itself — fine-tune a small open model into a specialist that's cheaper, faster, and more consistent than prompting a big general one. Across four lessons you'll go brief → data → LoRA training run (L12) → evaluation vs. the baseline (L13) → serving (L14).

This first lesson owns the part that decides whether the whole project succeeds: the brief (is fine-tuning even the right tool, and for what?) and the data. Here's the law that governs everything downstream: a fine-tuned model is a mirror — it becomes exactly what you show it. The training run in L12 is almost mechanical; the dataset is the product, and curating it well is the actual skill. Garbage in, garbage out — but clean, well-formatted, leakage-free data in, and a 1-billion-parameter model can beat a prompted 8-billion one on your task.

You'll do three things:

• Decide — confirm fine-tuning beats prompting and RAG for this task (it changes how the model behaves, not what it knows).
• Write the brief — a concrete, narrow, high-volume task with a crisp success metric.
• Curate the dataset — source, synthesize, clean, dedup, decontaminate, split, and format it into the JSONL the trainer expects, with a held-out test set kept sacred for L13.

Why this is a senior skill. Anyone can run a training script. Knowing when to fine-tune (most teams shouldn't), and being able to engineer a dataset that's diverse, clean, and leakage-free, is what separates a model that ships from one that quietly memorizes its training set. We use the course stack: Claude claude-sonnet-4-6 as the teacher that generates and judges data, and a small open model as the student we'll fine-tune in L12.

Scope: this lesson owns the brief + the dataset. Training → L12 (LoRA Fine-Tuning Run).

Before you curate a single example, earn the right to fine-tune. Most teams reach for it too early. The rule of thumb: start with prompting, scale with RAG, and only then specialize with fine-tuning — in that order, because each step costs more and is harder to change.

The decisive question: "is this a knowledge problem or a behavior problem?" If the model needs to know something (your policies, today's prices), that's RAG — don't fine-tune to teach facts (it memorizes poorly and goes stale). If the model needs to behave a certain way — always emit valid JSON in your schema, write in your brand voice, follow a rigid reasoning pattern, handle a niche task that would need an ever-growing prompt — that's fine-tuning. The classic tell: "if you can solve it with 10 examples in the prompt instead of 10,000 in a training set, you don't need fine-tuning." And the classic payoff: a small specialized model that does one narrow job reliably, cheaply, and fast — often at a fraction of a frontier model's cost and latency. (You can also do both: fine-tune for behavior and RAG for knowledge.)

Here's our project, and it's a textbook fine-tuning case born straight out of Projects 1–2:

Brief. Our support stack (the RAG assistant + agent) handles tickets beautifully with Claude — but running a frontier model on every one of thousands of daily tickets is slow and expensive. We'll distill the triage behavior into a small, fast, cheap specialist: given a ticket, output strict JSON — {category, priority, team} — in our fixed taxonomy, as the high-volume first pass, escalating only the hard ones to the big model.

Check it against the decision:

• Behavior, not knowledge — the taxonomy is fixed; the job is to reliably emit valid JSON in our schema. That's a behavior/format problem → fine-tuning. ✓
• Narrow + high-volume — one well-defined task, run millions of times, where latency and cost dominate. A small tuned model is ~20× cheaper and faster than prompting a frontier model per ticket. ✓
• Abundant data — we already have production logs of good triage decisions (the data flywheel from L10 — Ship It), and Claude can label more on demand. ✓

Success metric (so L13 can judge it): on a held-out test set, the tuned model must match the Claude baseline's triage within a target accuracy (say ≥ 95% category, ≥ 90% priority), emit 100% schema-valid JSON, and do it at a fraction of the cost/latency. A sharp brief with a measurable target is what keeps the dataset honest — every curation choice serves this metric.

You can't fine-tune a closed model like Claude — fine-tuning needs open weights. So the project is a distillation: a strong teacher (Claude claude-sonnet-4-6) generates and grades the training data, and we fine-tune a small open student to imitate it. The recipe:

• Method — SFT (Supervised Fine-Tuning). Train on (input → desired output) pairs so the student learns the mapping. (Preference methods like DPO come later in a model's life; SFT is the workhorse for a task like this.)
• Technique — LoRA / QLoRA. Don't update all the weights — train tiny low-rank adapters (parameter-efficient), so it fits on one affordable GPU. The mechanics are L12's job; here just know the student is small and the tuning is cheap.
• Student — a small open model (e.g. an 8B or even a 1–4B instruct model). Counter-intuitively, fine-tuning matters more than base-model size: a well-tuned small model routinely beats a prompted larger one on a narrow task — that's the entire reason this project is worth doing.

Everything above is downstream of the data. A perfect training recipe on a sloppy dataset gives a sloppy model; a clean dataset makes the rest almost mechanical. So the real work — this lesson — is the data.

Internalize this and you'll out-engineer most teams: for a fine-tune, the dataset — not the training code — is the artifact you're really building. The model becomes a mirror of its training data, so every property you want in the model has to be a property of the data:

• Want valid JSON every time? Every training output must be valid JSON in the schema — one malformed example teaches the model that malformed is acceptable.
• Want it to handle refunds, outages, and feature requests? All of those categories must be covered in the data — a fine-tuned model is terrible at what it never saw.
• Want consistency? The labels must be consistent — contradictory examples teach the model to be random.

And the counter-intuitive law: quality beats quantity, by a lot. A small, meticulously curated set (think 500–5,000 clean examples for a narrow task) outperforms a large noisy one — duplicates and mislabels actively teach the wrong thing. The workflow is iterative, just like the eval loop from L4 (Building the Evaluation Harness): curate a small batch, train, look at the failures, fix the data, repeat. You're not collecting data — you're engineering it.

Curation ends in a precise format, because the trainer is picky. The universal choice is JSONL (one JSON object per line) in the chat / messages schema — the same system / user / assistant shape you've used all course. Each line is one training example:

Three format details that quietly decide whether training works: (1) use the model's own chat template — the trainer renders these messages with the student model's template so it sees one consistent format; (2) loss-mask the prompt — compute the training loss only on the assistant turn, so the model learns to produce the answer, not to parrot the question; (3) append EOS so it learns to stop. Frameworks like TRL's SFTTrainer handle (1)–(3) for you when your data is in this messages format — which is exactly why we format to it. (Alpaca-style {instruction, input, output} works too, but chat messages is the most portable.)

Where do the examples come from? Four sources, best mixed together:

1. Production logs (highest value). Real tickets your system already triaged well — the data flywheel from L10 (Ship It). Real inputs, real distribution, free. This is your gold seed.
2. Expert-labeled seed. A few hundred examples a human (or you) labels by a clear guideline — the ground truth that anchors everything.
3. Distillation from the teacher. Have Claude label more tickets, capturing its triage behavior into (ticket → JSON) pairs. This is how a small student inherits a frontier model's skill.
4. Synthetic generation. Have the teacher invent realistic tickets for under-covered categories (rare priorities, edge cases), then filter them.

Distillation + a judge filter, on the course stack:

Two cautions the research is loud about. Mix your teachers / anchor on real data — a dataset generated by a single model collapses toward sameness (low diversity); seed it with real logs and (optionally) more than one generator to keep it diverse. And always filter — an unfiltered synthetic set is worse than a smaller judged one; every example that survives must be valid and in-taxonomy. Distillation is powerful, but it's garbage-amplification without the judge.

Raw examples aren't a dataset. They flow through a pipeline, and the order of the stages is the whole skill — get it wrong and you silently poison your evaluation. The canonical order:

spec → build eval set → source seed → synthesize → clean → dedup → decontaminate → split → format → version

Two ordering rules carry everything:

• Build the eval set FIRST, before any synthesis — and keep it sacred. Carve a real, held-out test set before you generate a single synthetic example, so synthetic data can never leak into your yardstick. L13's verdict is only trustworthy if the test set was untouched.
• Dedup and decontaminate BEFORE you split. Otherwise the same example (or a near-duplicate) lands in both train and test, and your eval score is inflated by memorization — the model looks great and fails in production.

The cleaning core — dedup and decontamination:

Around those two, the clean stage drops junk (too short/long, wrong language, PII, toxicity, and — critically — schema-invalid outputs), and the format stage applies the chat template + loss-mask + EOS from the previous section. The interactive below lets you assemble this pipeline yourself and watch a live checker catch the ordering bugs — it's the fastest way to make the two rules stick.

Here's the whole pipeline as one script — it produces the three JSONL files L12 will train on, in the right order, with the test set carved first and leakage removed before the split:

That's a reproducible, leakage-safe dataset: train.jsonl and val.jsonl to fit and monitor the model, and a sacred test.jsonl you won't touch until L13 (Evaluation vs Baseline). Commit the data and a short dataset card (where each example came from, sizes, date, taxonomy version) so the whole run is reproducible. Now you can hand these files to the trainer in L12.

Before L12, your dataset should be engineered, not collected:

• A sharp brief — a narrow, high-volume task where fine-tuning beats prompting/RAG (behavior/format, not knowledge), with a measurable success metric.
• A sacred held-out test set, carved first (before any synthesis), human-checked — reserved for L13.
• Sourced + synthesized examples — real logs + expert seed + teacher-distilled + filtered synthetic for coverage; every output valid and in-taxonomy.
• Cleaned, deduped, decontaminated — junk/PII/schema-invalid dropped; exact + near-dup removed; no eval-set overlap — all before the split.
• Split + formatted — train / val / test as chat-messages JSONL, ready for the chat template + loss-mask + EOS.
• Versioned — data committed with a dataset card (sources, sizes, taxonomy version) → reproducible.

If you can point at three JSONL files and prove the test set never saw a synthetic or duplicate example, you've curated a dataset you can trust — and L12 becomes the easy part.

Data curation lives or dies on stage order, so this builder makes you assemble the pipeline yourself and validates it live. Click the ten stages from the palette into the order you'd run them, reorder with the arrows, set the target size and train/val/test ratio, and watch the order checks light up.

• Try the obvious-but-wrong order first — synthesize before building the eval set, or split before dedup — and watch the checker flag the exact leakage bug.
• Fix it until you reach ship-ready: a leakage-safe, reproducible dataset and three JSONL files.
• Stuck? Hit canonical order to reveal the reference pipeline, then read why each rule exists.

Notice the two rules the checker is really enforcing. One: the eval set is built first and kept sacred — synthetic data must never touch your yardstick. Two: dedup + decontaminate happen before the split — or the same example lands in train and test and your L13 score is a memorization mirage. Order is the craft.

Reason these out, then check against the builder and the code.

1. A teammate wants to fine-tune the model so it "knows our latest pricing." Why is that the wrong tool, and what should they use instead?

2. You build a beautiful synthetic dataset, then carve your test set out of it at the end. What's wrong, and what does it do to your L13 evaluation?

3. Your dataset has 50,000 examples but the model still emits invalid JSON ~5% of the time. You discover ~3% of training outputs were themselves malformed. What's the law at work, and what's the fix?

4. Why must dedup and decontamination run before the train/test split, not after?

5. Why distill from Claude (the teacher) at all, instead of just using Claude in production for triage?

Answers.

1. "Knowing the latest pricing" is a knowledge problem, and fine-tuning changes behavior, not knowledge — it memorizes facts poorly and they go stale the moment prices change (you'd have to retrain). Use RAG: put the current pricing in context at query time. Fine-tune for how to answer, RAG for what is true.

2. You contaminated your yardstick. If the test set is carved from synthetic data, you're measuring how well the model reproduces the teacher's synthetic examples, not how it does on real tickets — and synthetic examples may have leaked into training too. Build the eval set FIRST, from real, held-out data, before any synthesis touches it.

3. The law: a fine-tuned model is a mirror — ~3% malformed training outputs teach it that malformed is ~3%-acceptable, so it reproduces the defect. Quantity can't fix it (50k examples, still broken). The fix is data quality: filter every output to be schema-valid in the clean stage; the defect vanishes when the data does. Quality beats quantity.

4. Because a split after dedup/decontam can still place an example and its near-duplicate (or an eval-overlapping row) into different splits — the same content in train and test. The model then memorizes it and scores artificially high on the test set, hiding real-world failure. Dedup + decontaminate first, then split, so train and test are genuinely disjoint.

5. Cost and latency at volume. Claude triages superbly but is too slow and expensive to run on every one of thousands of daily tickets. Distilling its behavior into a small open student gives you a model that's ~20× cheaper and faster for the high-volume first pass — escalating only the hard cases to the teacher. You keep the teacher's quality at the student's price. (And you can only LoRA-tune an open model — you can't fine-tune Claude's weights.)

• "Fine-tune to add knowledge." → Fine-tuning changes how the model behaves, not what it knows. For facts/freshness use RAG; fine-tune for format, style, and behavior on a narrow task.
• "Fine-tuning is the first thing to try." → It's the last. Prompt → RAG → fine-tune, in that order. If 10 prompt examples solve it, you don't need 10,000 training examples.
• "The training script is the hard part." → The dataset is. The model is a mirror of its data; L12's run is almost mechanical once the data is right.
• "More data is better." → Quality beats quantity. A clean, diverse 500–5k set beats a noisy 50k one — duplicates and mislabels actively teach the wrong thing.
• "Synthetic data is free scale." → Only with a judge filter and a real-data anchor. Unfiltered, single-teacher synthetic data collapses toward sameness and amplifies errors.
• "Split first, clean later." → Dedup + decontaminate BEFORE the split, or the same example leaks across train/test and your eval is a memorization mirage.
• "Build the eval set from whatever's left." → Build it FIRST, from real held-out data, and keep it sacred — synthetic data must never touch your yardstick.
• "Any JSON-ish format will do." → Use chat messages JSONL, the model's chat template, loss-mask the prompt, and append EOS — formatting is what makes a row actually trainable.

• Earn the right to fine-tune. Prompt → RAG → fine-tune. Fine-tuning changes behavior/format/style, not knowledge — use it for a narrow, high-volume task where consistency and cost/latency matter.
• The dataset is the product. A fine-tuned model is a mirror of its data; quality beats quantity (500–5k clean, diverse, consistent examples beat a noisy 50k). Curate iteratively, like the L4 eval loop.
• Distill on the course stack. Use Claude claude-sonnet-4-6 as the teacher (generate + judge data) for a small open student; mix in real logs (the L10 flywheel) and filter everything.
• Order is the whole craft: spec → eval set first (sacred) → source → synthesize → clean → dedup → decontaminate → split → format → version. Build the eval set before synthesis; dedup + decontaminate before the split.
• Format precisely — chat messages JSONL, the model's chat template, loss-mask the prompt, EOS to stop. Output three files: train / val / sacred test, plus a dataset card for reproducibility.
• Next — L12 (LoRA Fine-Tuning Run): with a clean dataset in hand, the training run is the easy part — you'll fit LoRA/QLoRA adapters on the small student and watch the loss curves.