Cleaning, Dedup & Formatting

You've sourced (L196), generated (L197), and maybe distilled (L198) a pile of examples. It is not a training set yet — it's raw material full of junk, duplicates, PII, leaks, and the wrong format. This lesson is the final pass that turns it into something you can actually train on: cleaning, deduplication, and formatting.

Think of it as the last gate — an ordered funnel where every bad row either gets fixed or dropped before it can teach the model the wrong thing. It's unglamorous and it's where careless teams lose: a forgotten eval leak inflates your scores into a lie, and a wrong chat template means the model never learns to stop. Three jobs, in order: clean (drop the junk), dedup (remove the repeats and the eval leakage), format (wrap it the way the trainer expects). Get this right and your L195 "good dataset" finally becomes a training-ready one.

The whole pipeline follows one design principle: run the cheap, aggressive filters first, and save the expensive operations for last — so only already-clean data reaches the costly steps. There's no point running a GPU-heavy semantic-dedup pass over rows that a one-line length check would have thrown away.

So the order is roughly:

1. Heuristic cleaning (length, language, format, toxicity) — milliseconds per row.
2. PII scrub — cheap regex + NER.
3. Exact dedup — a hash lookup.
4. Near-dup (MinHash) — fast, scalable fuzzy matching.
5. Semantic dedup — embeddings + clustering (the expensive one).
6. Decontamination — n-gram overlap against your eval set.
7. Formatting — the chat template, last.

Heuristic, rule-based filters are also preferred over ML-based ones where possible — they're transparent and don't inject model bias. You'll feel this order in the lab (it even warns you if you run semantic dedup before the cheap filters).

Cleaning is quality-filtering the rows. The standard heuristic screens:

• Length — drop examples that are too short (e.g. < ~20 tokens — no signal) or pathologically long (often a scraping artifact).
• Language — a fast classifier (e.g. FastText) drops documents below a confidence threshold for your target language.
• Format / encoding — broken Unicode, leftover HTML, truncated responses, empty fields.
• Toxicity / illegal — remove harmful or non-compliant content.

Then the privacy pass from L196 (Sourcing & Annotating Data): PII scrub — redact names, emails, phone numbers, and card numbers in the free text, not just labeled fields (it hides in the body of a ticket or transcript). Finally, optional model-based quality scoring (the LLM-as-judge from L197 (Synthetic Data Generation)) drops low-quality rows. Remember the order: heuristics are cheap, the judge is expensive — filter with rules first, then spend judge tokens only on what survives.

Duplicates add tokens, not signal — they waste compute, bias the model toward the repeated examples, and (worst) cause train/eval leakage. But "duplicate" has three levels, and you need all three because each catches a different kind of repeat:

1. Exact dedup — normalize (lowercase, collapse whitespace) and hash; drop identical rows. Cheap, first.
2. Near-duplicate (fuzzy) — MinHash + LSH over n-gram shingles (e.g. 5-grams). Fast and scalable; catches rows that differ by a word or two ("reset my password?" vs "reset my password??"). It's lexical — driven by surface overlap.
3. Semantic dedup — embed each row with a sentence encoder (e.g. all-MiniLM-L6-v2), then cluster (SemDeDup / k-means) and drop near-identical meaning even when the wording is totally different ("reset my password" vs "recover my login"). A cosine threshold around 0.8 is common.

Strong pipelines chain all three: exact → MinHash → SemDeDup. The MinHash pass is cheap and catches the obvious near-copies; the semantic pass is expensive but catches the paraphrases MinHash misses. Which sets up the step everyone forgets.

Here's the one that silently ruins evaluations: train/eval contamination. If a training example is also in your eval/test set (or a public benchmark like MMLU or GSM8K), the model memorizes it and your eval score goes up — but the model isn't actually better, you're just measuring memorization, not skill. Your numbers become a lie you tell yourself.

The trap: exact dedup does NOT catch this. Paraphrased questions, solution fragments, distilled samples, and discussion-board copies are all semantically close to the eval items but lexically different — they sail past a hash check while still leaking the answer. So decontamination is its own step: scan training rows for n-gram overlap (the practical standard) — and ideally semantic overlap — against your held-out eval and any benchmarks you report on, and remove the matches.

It's a balance: over-aggressive filtering creates false positives that throw away good training data; incomplete filtering leaves leakage that inflates scores. But the rule is non-negotiable: your training set and your eval set must not overlap — decontaminate before you trust a single eval number (this is what makes the Container 4 evals honest).

The data is clean and unique — now it has to be in the exact shape the trainer (and model) expect. This is where the #1 silent fine-tuning bugs live, because the model imitates the format exactly (the L195 mirror). Three things must be right:

• The chat template. Modern models ship a Jinja2 chat template in their tokenizer config (ChatML-style: <|im_start|>role … <|im_end|>, with the model's special tokens). You must wrap every row in the model's own template via apply_chat_template — using the wrong template (or none) teaches the model a format it'll never see at inference, and quality collapses. Be consistent: one template, every row.
• Loss masking (train on completions only). Compute the loss only on the assistant's response tokens, masking the prompt/instruction. You want the model to learn to generate answers, not reproduce prompts. This is the common default and TRL does it for you.
• The EOS token. Every example must end with EOS so the model learns to stop. Forget it and your fine-tuned model never stops generating — a classic, baffling bug.

In TRL this is mostly handled for you if you format correctly:

Here's the whole funnel as a pipeline — cheap filters, three dedup levels, decontamination, then format:

Run the funnel and watch the rows fall. Start with everything off — 12 messy rows. Now turn stages on one at a time and watch each defect get caught: the too-short row dies at length, the French row at language, the exact copy at exact dedup, the reworded one at MinHash, the same-meaning one at semantic dedup. Then the two that matter most: leave Decontaminate off and watch the eval-leak row survive (your scores will lie); leave Format off and the verdict refuses to call it training-ready (no template / EOS). Hit run full pipeline to see a clean, deduped, formatted set.

Notice the order tip: run semantic dedup with the cheap filters off and it nags you — clean first, then spend compute. That's the funnel principle in one interaction.

This is the step that separates a dataset that looks fine from one that actually trains a good model. Skip dedup and you waste compute and skew the model toward repeated examples. Skip decontamination and every eval number you report is inflated by memorization — you'll ship a model you think is great and discover in production that it isn't. Botch the formatting and you get the two most baffling beginner bugs in all of fine-tuning: a model that learned the wrong template, or one that never stops generating. Mastering this funnel is what makes the difference between "I fine-tuned a model" and "I fine-tuned a model that works and I can trust my evals." It's the bridge to the capstone — L200 (Hands-On: Build a Fine-Tuning Dataset) — where you assemble everything from this section end to end.

Find the traps in the Cleaning Pipeline:

1. The dedup ladder. Turn on exact dedup only — does it catch the reworded row (#5) or the same-meaning row (#6)? Now add MinHash, then semantic. Why do you need all three?
2. The silent leak. Get every dedup stage on but leave Decontaminate off. Which row survives, and why is exact dedup not enough to catch it? What does its survival do to your eval scores?
3. Not training-ready. Clean and dedup everything but leave Format off. Why does the verdict still refuse to ship it — name the three things the format step does.

Bonus: you ran the expensive semantic dedup but forgot the length and language filters. What does the lab nag you about, and why is that the wrong order?

• "Exact dedup is enough." No — it misses near-duplicates (MinHash) and paraphrases (semantic), and crucially it does not catch eval leakage.
• "Decontamination is just deduplication." Different goal: dedup removes repeats within the training set; decontamination removes overlap with the eval set — the thing that inflates your scores.
• "Formatting is trivial." It's the #1 silent bug: wrong chat template, no loss mask, or missing EOS quietly wreck the run (or make the model never stop).
• "Order doesn't matter." Run cheap filters first — don't waste expensive semantic dedup / judge passes on rows a length check would have dropped.
• "Clean once at the end." Cleaning is the last gate, but quality starts upstream (L195–L198); this pass enforces it, it can't create quality that was never sourced.

• Cleaning is the last gate, run as an ordered funnel — cheap, transparent heuristics first, expensive model-based steps last.
• Clean: drop too-short / off-language / malformed / toxic rows, scrub PII from free text, then optional model-based quality scoring.
• Dedup at three levels: exact (hash) → near-dup (MinHash+LSH, lexical) → semantic (embeddings, paraphrases). Each catches a different repeat.
• Decontaminate against your eval/benchmarks — exact dedup won't catch leakage, and leakage inflates scores via memorization. Train and eval must not overlap.
• Format with the model's own chat template (apply_chat_template), loss-masked to the response, ending in EOS — the three most common silent fine-tuning bugs.

Next: L200 — Hands-On: Build a Fine-Tuning Dataset — the capstone that assembles sourcing, synthesis, distillation, and this cleaning funnel into one real, training-ready dataset.