Hands-On: Build a Fine-Tuning Dataset

Introduction

This is the capstone of the section — where every lesson since L195 (Data Quality, Coverage & Quantity) snaps together into one repeatable pipeline that produces a real, training-ready dataset. Not theory: the actual end-to-end recipe you'll run to go from "I want to fine-tune X" to three versioned JSONL files (train / val / test) you hand straight to TRL's SFTTrainer.

The reframe that makes it click: a fine-tuning dataset is not a file you download — it's a product you engineer. It has a spec, a build pipeline, quality gates, versions, and a reproducible process. Each earlier lesson was one stage of that pipeline; this lesson is the assembly line that runs them in the right order. And as you'll see, order is everything — two sequencing mistakes silently ruin datasets that look perfect.

An infographic titled 'Hands-On: Build a Fine-Tuning Dataset', the capstone of section three on dataset engineering, which assembles every earlier lesson into one repeatable end-to-end pipeline that produces a training-ready dataset. The framing is that a fine-tuning dataset is not a file you download but a product you engineer through ten ordered stages. First, spec the task: the output format, the label taxonomy, the production distribution to cover, and the target size, drawn from the quality, coverage, and quantity axes. Second, and critically, build the evaluation set first by carving out a real held-out test set before any synthetic data is generated, so the yardstick stays clean and uncontaminated. Third, source the seed from production logs and an expert-written seed of roughly five hundred to a thousand examples with a clear annotation guideline. Fourth, synthesize to expand and fill coverage gaps with a teacher model and a judge filter, or distill from a larger model. Fifth, clean with cheap heuristic filters for length, language, format, and toxicity, plus a PII scrub. Sixth, deduplicate at three levels: exact, MinHash near-duplicate, and semantic. Seventh, decontaminate by removing any training row that overlaps the evaluation set, because exact dedup does not catch leakage. Eighth, split into train, validation, and test, after deduplication, using ratios such as eighty ten ten or ninety five five, stratified for small or imbalanced data, where training teaches, validation tunes, and the test set stays sacred. Ninth, format every row in the model's own chat template, loss-masked to the response and ending in the end-of-sequence token. Tenth, version the dataset and write a dataset card documenting provenance and license so the run is reproducible. Two ordering rules carry the whole section: build the eval set before synthesis, and deduplicate and decontaminate before the split, or the same example leaks across train and test. The takeaway is that the dataset is engineered in order, then handed to a trainer.

The Pipeline: Ten Stages, In Order

Here's the whole assembly line. Each stage is a lesson you've already learned; the skill now is running them in sequence:

Spec the task — output format, label taxonomy, the production distribution to cover, and the target size (the L195 axes). Write the success criteria + which evals you'll run.
Build the eval set FIRST — carve a real, held-out test set before you generate anything.
Source the seed — production logs + an expert seed (~500–1k) with a guideline + measured IAA (L196).
Synthesize / distill — expand the seed and fill coverage gaps with a teacher + judge (L197), or compress a bigger model (L198).
Clean — cheap heuristics (length, language, format, toxicity) + PII scrub (L199).
Dedup — exact → MinHash → semantic (L199).
Decontaminate — remove any row overlapping the eval set (L199).
Split — train / val / test (after dedup).
Format — the model's chat template + loss-mask + EOS (L199).
Version + dataset card — document provenance/license; make it reproducible.

Most of these you've met. The new ideas in this lesson are the two that people get wrong: when to build the eval set, and when to split.

Rule 1: Build the Eval Set FIRST

The most important sequencing decision in the whole pipeline: carve out your held-out evaluation/test set before you generate or augment a single example.

Why first? Because your eval set is the yardstick that tells you whether the fine-tune actually worked (the Container-4 evals). If you build the dataset, then split off a test set, your synthetic data (L197) and near-duplicates have already mixed in — and your "held-out" set is quietly contaminated with paraphrases of the training data. You'll get a great eval number that's really just memorization (the L199 leakage problem, baked in from the start).

So the discipline is: take a sample of real, representative data, set it aside as the test set, and treat it as sacred — no synthetic expansion, no augmentation, never trained on. "Build your evaluation framework before you train" is one of the strongest predictors of a fine-tune that actually ships. Everything else — seed, synthesis, cleaning — happens to the other pile.

Rule 2: Dedup & Decontaminate BEFORE You Split

The second sequencing trap is when you split into train / val / test. The rule: deduplicate and decontaminate FIRST, then split.

If you split before dedup, the same example (or a near-duplicate) can land in both the training set and the test set — instant cross-split leakage, and your test score is a lie. Dedup first, and every example exists once, so it can only fall on one side of the split.

Then split with sane ratios:

80/10/10 or 90/5/5 (train/val/test) for typical sets; 98/1/1 when you have a lot of data.
Use a stratified split for small or imbalanced data, so every split has the same class/label distribution.

And know the three roles — this is the thing beginners blur:

Train — the model learns from it.
Validation — you watch it during training (early stopping, hyperparameters — L194). It's how you tune.
Test — touched once, at the very end, for an unbiased number. If you tune against the test set, it stops being unbiased — that's what the validation set is for. Keep test sacred.

Version It: The Dataset Card

A dataset you can't reproduce is a liability — six weeks later you won't remember which seed, which teacher, which filters produced it, and you won't be able to explain a regression. So the final stage is documentation + versioning:

A dataset card (on the HF Hub, it's the README.md with YAML frontmatter) — record the provenance (where each source came from), the license, the size and splits, and how it was built (seed → synth → clean steps).
Version it — tag the dataset (e.g. v1.0) so a training run pins an exact dataset, and you can diff v2 against v1 when you iterate.
Optionally push_to_hub to store and share it.

This is what turns "I fine-tuned a model once" into a repeatable, auditable process — and it ties back to the L196 legal layer (you can prove your data's provenance and rights).

In Code: The Whole Build

Here's the pipeline as one script — note the order: eval set carved first, dedup + decontaminate before the split, format last.

from datasets import load_dataset, concatenate_datasets

# 1-2. SPEC, then carve the EVAL SET FIRST from REAL data (never synthesized/augmented).
real = load_dataset('json', data_files='real_logs.jsonl', split='train')
real = real.shuffle(seed=42)
test = real.select(range(500))            # held-out, SACRED - set aside before anything else
seed = real.select(range(500, len(real))) # the rest seeds the training pile

# 3-4. SOURCE + SYNTHESIZE to fill coverage gaps (L196/L197), then merge with the seed.
synth = generate_synthetic(seed, teacher='Llama-3.3-70B', judge_min=4)   # filtered
pool  = concatenate_datasets([seed, synth])

# 5-7. CLEAN -> DEDUP -> DECONTAMINATE (against the test set) - BEFORE the split.
pool = pool.filter(is_clean).map(scrub_pii)
pool = sem_dedup(minhash_dedup(exact_dedup(pool)))
pool = pool.filter(lambda r: not ngram_overlap(r, test, n=13))   # no train/test leak

# 8. SPLIT the cleaned pool (test already set aside) -> 90/5/5 here = train/val.
sp = pool.train_test_split(test_size=0.055, seed=42)
train, val = sp['train'], sp['test']

In Code: Format, Version & Hand Off

Then format with the tokenizer's chat template, write the JSONL, document it, and hand it to SFTTrainer — looping you right back to L194 (Fine-Tuning Hyperparameters & Tactics):

# 9. FORMAT with the model's OWN chat template (+ EOS); loss-mask handled by SFTTrainer.
def fmt(r): return {'text': tok.apply_chat_template(r['messages'], tokenize=False)}
train, val, test = train.map(fmt), val.map(fmt), test.map(fmt)
for ds, name in [(train,'train'),(val,'val'),(test,'test')]:
    ds.to_json(f'{name}.jsonl')

# 10. VERSION: a dataset card + push (provenance, license, splits) -> reproducible.
train.push_to_hub('me/support-sft', config_name='v1.0', split='train')   # + val/test

# HAND OFF to training (back to L194 for the hyperparameters).
from trl import SFTTrainer, SFTConfig
SFTTrainer(model=model, train_dataset=train, eval_dataset=val,
           args=SFTConfig(completion_only_loss=True, eval_strategy='steps')).train()
# Final, unbiased number comes from TEST - touched once, at the very end.

See It: The Dataset Builder

Assemble the pipeline yourself. Click the ten stages from the palette to add them in the order you'd run them, reorder with the arrows, and watch the order checker validate live. Try to break the two rules: put Synthesize before Build eval set (watch the contamination flag) or Split before Dedup/Decontaminate (watch the leakage flag). Set your target size and split ratio and see the train/val/test counts. Hit canonical order to reveal the reference pipeline.

Interactive: the Dataset Builder, the section capstone. The user assembles the ten-stage fine-tuning-dataset pipeline by clicking stages from a palette to add them in the order they would run, reordering with up and down controls, and removing them, with arrow connectors showing the flow. A live order-checker validates the sequence against the 2026 best practices: spec the task first, build the evaluation set before synthesizing so synthetic data never contaminates the yardstick, deduplicate and decontaminate before the split or the same row leaks across train and test, decontaminate at all because exact dedup misses leakage, format before versioning, and include a versioning step for reproducibility. A dataset-stats panel shows the target size, a selectable split ratio of eighty ten ten, ninety five five, or ninety-eight one one, the resulting train, validation, and test counts, and two badges for leakage-safe and reproducible. A ship-readiness verdict names the first ordering bug, and a load-the-canonical-order button reveals the reference pipeline so the user can compare their ordering to the correct one.

When every check is green you've got a leakage-safe, reproducible dataset — three JSONL files ready for SFTTrainer. That sequence is the section.

Why This Matters

This is the lesson that turns a pile of techniques into an actual capability: you can now take any fine-tuning problem and engineer the dataset end-to-end — spec it, build the eval first, source and synthesize, clean and dedup, decontaminate, split, format, and version — and hand off a dataset you can trust and reproduce. That's the difference between someone who's read about fine-tuning data and someone who can ship it. It also closes Section 3: Dataset Engineering — and with a real dataset in hand, you're ready for Section 4: Advanced Post-Training, where preference tuning (DPO and friends) takes a fine-tuned model from competent to aligned with what people actually prefer.

🧪 Try It Yourself

Build the pipeline and break it in the Dataset Builder:

Add all ten stages in your own guessed order (don't peek). How many order-checks pass? Then hit canonical order and compare — which stage did you misplace?
Cause the contamination bug. Put Synthesize before Build eval set. What does the checker say, and why does that ruin your eval numbers?
Cause the leakage bug. Put Split before Dedup and Decontaminate. What goes wrong, and which split is compromised?

Bonus: for a 1,000-example set, which split ratio would you pick and why? What's the one split you must never tune against — and which split is for tuning?

Mental-Model Corrections

"Build the dataset, then split off a test set." Backwards — carve the eval set FIRST from real data, before synthesis, or it's contaminated.
"Split, then dedup each set." Also backwards — dedup + decontaminate before splitting, or the same row leaks across train and test.
"Validation and test are the same thing." No — validation is for tuning during training (early stopping, hyperparameters); test is touched once for an unbiased final number. Tune against test and it's worthless.
"A dataset is just the training file." It's a product: spec, pipeline, splits, a dataset card, and a version — so it's reproducible and auditable.
"More data always helps the splits." Only clean, deduped, uncontaminated data — and the test set stays small and sacred no matter how big the train set gets.

Key Takeaways

A fine-tuning dataset is a product you engineer through a repeatable 10-stage pipeline: spec → eval-first → source → synthesize → clean → dedup → decontaminate → split → format → version.
Rule 1 — build the eval set FIRST (real, held-out, sacred) before any synthesis, or your yardstick is contaminated.
Rule 2 — dedup & decontaminate BEFORE the split, or the same example leaks across train/test.
Splits: 80/10/10 or 90/5/5 (stratified for small/imbalanced). Train teaches, val tunes (L194), test is touched once for an unbiased number.
Version it — a dataset card (provenance, license, splits) + a version tag makes the run reproducible; then hand the JSONL to SFTTrainer.

That closes Section 3: Dataset Engineering. Next, Section 4 — Advanced Post-Training: turning a fine-tuned model into one aligned with human preferences (DPO and beyond).