Data Quality, Coverage & Quantity

Introduction

You can have the perfect method (L193 — LoRA & QLoRA Explained) and the perfect knobs (L194 — Fine-Tuning Hyperparameters & Tactics) and still ship a useless model — because the thing that decides whether any of it matters is the data. This lesson opens Section 3: Dataset Engineering, and it starts with the foundation: what makes a fine-tuning dataset good.

Here's the mental model that reframes everything: a fine-tuned model is a mirror, not a student. It doesn't reason about your examples — it imitates the distribution you show it: your format, your label conventions, your tone, and (ruthlessly) your mistakes. So the dataset isn't an input to the process — it is the process. We'll judge it on three axes that multiply together — quality, coverage, and quantity — and you'll come away knowing exactly which one to fix first when a fine-tune underperforms.

An infographic titled 'Data Quality, Coverage & Quantity', the lesson that opens section three on dataset engineering, the part of fine-tuning that decides whether any of the hyperparameters matter. Its central metaphor is that a fine-tuned model is a mirror, not a student: it imitates the distribution you show it, absorbing your format, your label conventions, and your mistakes, rather than reasoning about them, which is why a curated dataset is the single biggest lever on the final model. The dataset is judged on three axes that multiply together, so any one near zero collapses the result. The first axis is quality: every example should be correct, clean, and consistent, because a wrong label is a wrong lesson and inconsistent formatting teaches the model to reproduce the variation instead of the task. The landmark evidence is LIMA, which fine-tuned a sixty-five-billion-parameter model on only one thousand carefully curated instruction-response pairs and matched far larger noisy sets, and the repeated finding that a few thousand curated examples beat fifty thousand machine-generated ones. The second axis is coverage: the training distribution must match the production distribution, spanning the full range of inputs the model will actually see, including the edge cases such as missing fields, rare formats, and ambiguous inputs, because the model only handles what it was shown and gaps in coverage become predictable production failures. Coverage has two parts, semantic breadth across task families and information depth within each example, and diversity is a key driver of success. The third axis is quantity, which matters least but has a floor and a plateau: roughly fifty to a hundred examples can begin to move the needle, five hundred to two thousand give noticeable task improvement, and five to ten thousand drive meaningful behavioral change, with the floor rising as task complexity grows from classification to extraction to generation to reasoning, and gains flattening past the plateau. The priority order is quality first, then coverage, then quantity, captured by the rule garbage in, garbage out: no amount of hyperparameter tuning rescues a flawed dataset. The takeaway is to fix quality and coverage before you scale quantity, and to make the training set a faithful, clean mirror of production.

The Big Idea: The Model Is a Mirror

Internalize this before anything else: the model becomes the average of what you show it. Fine-tuning doesn't teach the model to think about your examples — it shifts its weights to reproduce them. That single fact explains every rule in this lesson:

Show it a consistent format → it locks onto that format. Show it an inconsistent one → it learns to reproduce the inconsistency.
Show it correct answers → it imitates them. Show it a wrong label → that's a lesson, not a typo; it learns the wrong answer.
Show it only common inputs → it handles common inputs. The edge cases you skipped simply aren't in the mirror, so it has nothing to reflect when they show up in production.

This is why "garbage in, garbage out" is the most important sentence in fine-tuning — and why no amount of L194 hyperparameter tuning can rescue a flawed dataset. The leverage is upstream, in the data. The three axes below are just the three ways a mirror can fail you.

Axis 1 — Quality: Every Example Is a Lesson

Quality is whether each example is correct, clean, and consistent. Because the model imitates exactly what it sees, three failures here are poison:

Correctness. A factually wrong or mislabeled example isn't noise the model averages away — it's a lesson that pulls the model toward the wrong answer. Even a small fraction of mislabeled rows visibly degrades behavior.
Consistency. This one surprises people. If your label is "billing" in one row, "Billing." in another, and "billing issue" in a third, the model learns to reproduce the variation, not the task. Inconsistent formatting, casing, or response style teaches the model to match the noise. Pick one convention and never drift.
Cleanliness. Typos, broken encoding, duplicated rows, and truncated responses all lower the ceiling (you'll clean these systematically in L199 — Cleaning, Dedup & Formatting).

The evidence is striking. LIMA ("Less Is More for Alignment") fine-tuned a 65B model on just 1,000 carefully curated instruction–response pairs and rivaled models trained on far more — and a few thousand curated examples beat the 50,000-example machine-generated Alpaca set. The lesson isn't "use little data"; it's that quality dominates, and a small impeccable set outperforms a large sloppy one.

Axis 2 — Coverage: Match the Production Distribution

Coverage asks: does the training set span the full range of inputs the model will actually see in production — not just the easy center? This is the axis people most often get wrong, because it's invisible until the model fails.

The rule is simple: the training distribution must match the production distribution. If 8% of your real traffic is "customer is angry and the order ID is missing," but that case is 0% of your training data, the model has never seen it in the mirror — and it will fail on exactly the inputs that matter most. Under-representing edge cases is the #1 source of production unreliability.

Think of coverage as two sub-dimensions:

Semantic breadth — how many kinds of input you cover: common queries, long/multi-part inputs, and the edge cases (missing fields, rare formats, ambiguous or adversarial inputs, out-of-scope requests and how to refuse them). Diversity is a key driver of success — more varied tasks even let you use fewer examples per task.
Information depth — how rich each example is within its category.

The practical move: build your coverage map from real production traffic (or your best estimate of it), then deliberately include the edge cases. A model is only as reliable as its least-covered important input — which is exactly what you'll feel in the lab's production stream.

Axis 3 — Quantity: The Floor and the Plateau

Quantity matters least of the three — but it's not nothing. It has a floor (too few and there's no signal to learn the pattern) and a plateau (past a point, more rows barely move the needle). Rough 2026 anchors for LoRA/QLoRA SFT:

~50–100 examples — can start to move the needle (style/format alignment).
~500–2,000 — noticeable task improvement.
~5,000–10,000 — meaningful behavioral change.

Two things shift these numbers:

Task complexity sets the floor. Easy tasks (classification, entity extraction) learn from fewer examples; hard, open-ended tasks (generation, summarization, multi-step reasoning) need more to cover the space.
Diversity beats raw count. Because of coverage, 1,000 diverse examples can beat 10,000 near-duplicates — duplicates add tokens, not signal (and can even leak between train and eval).

The shape to remember is a scaling curve: steep gains as you climb off the floor, then a plateau. Once you're on the plateau, the answer to "my model isn't good enough" is almost never more rows — it's better quality or wider coverage.

The Priority Order: Quality > Coverage > Quantity

The three axes multiply — predicted reliability ≈ quality × coverage × quantity — which means any one near zero tanks the whole model, and you cannot compensate for a weak axis by maxing another. Ten thousand rows of a noisy, narrow set is still a noisy, narrow model.

So fix them in order:

Quality first. Clean, correct, one consistent convention. This is the cheapest, highest-leverage win — and the model imitates your mistakes, so they're expensive to leave in.
Coverage second. Make the set a faithful sample of production, including the edge cases. Find the gaps before your users do.
Quantity last. Scale up — but only as far as the curve still pays, and never at the cost of quality or coverage.

One more guardrail from L187 (Fine-Tuning + RAG Together): fine-tuning data is great at teaching form, style, and behavior, but teaching the model new facts by example is unreliable and can even hurt factuality. Keep volatile knowledge in retrieval; use fine-tuning data to shape how the model responds.

What the Model Imitates: Keep Every Row Identical

Quality and consistency are concrete, not abstract. Here's the chat / messages format (the TRL SFTTrainer default) — and the exact shape the model will lock onto. The danger isn't the content; it's drift between rows.

# A fine-tune learns the EXACT shape you show it - keep every row identical.
# chat / 'messages' format (the TRL SFTTrainer default):
data = [
  {'messages': [
    {'role': 'system',    'content': 'You are a support classifier. Reply with ONE label.'},
    {'role': 'user',      'content': 'My card was charged twice for one order.'},
    {'role': 'assistant', 'content': 'billing'}]},
  {'messages': [
    {'role': 'system',    'content': 'You are a support classifier. Reply with ONE label.'},
    {'role': 'user',      'content': 'How do I reset my password?'},
    {'role': 'assistant', 'content': 'account'}]},
]
# DANGER: 'billing' vs 'Billing.' vs 'billing issue' across rows teaches the model
# to reproduce the VARIATION, not the task. Pick ONE convention and never drift -
# same system prompt, same label set, same response style, every single row.

A 60-Second Dataset Audit

Before you spend a GPU-hour, measure the three axes. This tiny audit catches the most common killers — duplicates, inconsistent labels, and coverage gaps — in seconds (the full pipeline is L199 — Cleaning, Dedup & Formatting).

# A 60-second quality + coverage audit BEFORE you train.
import collections, hashlib

def norm(s): return ' '.join(s.lower().split())

seen, dupes, labels, cov = set(), 0, collections.Counter(), collections.Counter()
for ex in data:
    user  = ex['messages'][1]['content']
    label = ex['messages'][-1]['content']
    h = hashlib.md5(norm(user).encode()).hexdigest()
    if h in seen: dupes += 1          # near-duplicate -> wasted (or leaking) rows
    seen.add(h)
    labels[label] += 1                # consistency: watch for 'billing' vs 'Billing.'
    cov[categorize(user)] += 1        # coverage: are the edge cases represented?

print('rows:', len(data), '| duplicates:', dupes)
print('label distribution:', dict(labels))   # spot inconsistent / skewed labels
print('coverage by category:', dict(cov))    # spot the GAPS vs production traffic
# Rule of thumb: dedup hard, ONE label convention, and make coverage MATCH production.

See It: The Dataset Studio

Now curate a set and watch it predict production reliability. Click the quality cards to inject defects (each one is a lesson the model learns). Toggle which input categories are in the coverage set and watch the production stream — uncovered inputs fail, live. Drag quantity along the scaling curve, and switch task complexity to move the floor. Watch the bottleneck verdict: because the axes multiply, the studio always points you at the one that's actually holding you back.

Interactive: the Dataset Studio. The user curates a fine-tuning set across the three axes and watches a predicted production-reliability score respond, learning that the model is a mirror and that quality and coverage gate the result while quantity alone cannot rescue a noisy or narrow set. For quality, six example cards can be clicked to cycle through clean, wrong-label, inconsistent-format, and noisy states, each defect dragging the quality bar down because the model imitates every example. For coverage, the user toggles which production input categories are present in the training set, from common queries to edge cases like missing fields, rare formats, and ambiguous inputs, and a live production stream connected by an arrow shows which incoming inputs are handled versus which fail as uncovered gaps. For quantity, a slider moves along a scaling curve with a data floor and a plateau that shift with a task-complexity selector spanning classification, extraction, generation, and reasoning. A reliability gauge equal to quality times coverage times quantity, a per-axis breakdown, and a binding-constraint verdict name the current bottleneck and its fix, making concrete that the three multiply so any one near zero tanks the model.

Try the experiment that proves the priority order: max out quantity to 50k while leaving a few cards mislabeled and the edge cases uncovered — and watch reliability stay low. Then fix quality and coverage at a fraction of the rows. Quality × coverage gate it; quantity can't buy your way out.

Why This Matters

Dataset engineering is where most fine-tuning projects are actually won or lost — and it's the part tutorials skip to get to the trainer.train() line. The engineer who treats the dataset as a faithful, clean mirror of production ships a model that works; the one who scrapes 50k rows and hopes ships one that memorizes noise and fails on edge cases. Mastering the three axes means that when a fine-tune underperforms, you don't flail at the hyperparameters — you ask "is it quality, coverage, or quantity?" and you know which to fix first. Everything in the rest of this section — L196 (Sourcing & Annotating Data), L197 (Synthetic Data Generation), L199 (Cleaning, Dedup & Formatting) — is in service of these three axes.

🧪 Try It Yourself

Diagnose three broken datasets in the Dataset Studio, and write the one-line fix for each:

The noisy set. Set quantity to 50k, full coverage, but flip three cards to wrong label. Why is reliability still capped, and which axis is the bottleneck?
The narrow set. All cards clean, quantity 10k, but only Common queries covered. What does the production stream show, and name two edge categories you'd add first.
The starved set. All cards clean, full coverage, switch to Reasoning, and drag quantity down to ~200. Why does reliability collapse even though quality and coverage are perfect?

Bonus: with the model-is-a-mirror idea, explain why a dataset of 1,000 diverse examples can beat 10,000 near-duplicates — what do the duplicates add, and what don't they?

Mental-Model Corrections

"More data is always better." Only up to the plateau, and only if it's clean and diverse. 10,000 noisy near-duplicates lose to 1,000 curated, varied examples.
"The model will average out a few bad examples." No — it imitates them. A wrong label is a lesson, and inconsistent format teaches it to reproduce the inconsistency.
"Coverage just means dataset size." No — it means matching the production distribution, especially the edge cases. A huge dataset that's all easy/common inputs has terrible coverage.
"I'll fix a weak dataset with better hyperparameters." You can't. Garbage in, garbage out — the leverage is upstream, in the data, not in L194's knobs.
"Fine-tuning data is how I teach the model new facts." Risky — it's great for form and behavior, unreliable for knowledge (and can hurt factuality). Keep facts in retrieval (L187).

Key Takeaways

The model is a mirror: it imitates the distribution you show it — your format, labels, and mistakes. The dataset is the process.
Three axes that multiply: quality (correct, clean, consistent — one convention, never drift), coverage (match the production distribution, including edge cases), quantity (a floor and a plateau; complexity raises the floor).
Quality dominates: LIMA's 1,000 curated examples rivaled far larger sets; a few thousand curated beat 50k noisy. Diversity beats raw count.
Priority order: quality > coverage > quantity. They multiply, so fix the weakest first — you can't buy your way out of a noisy or narrow set with volume.
Garbage in, garbage out: no hyperparameter rescues a flawed dataset, and fine-tuning data shapes form, not facts (keep knowledge in retrieval).

Next: L196 — Sourcing & Annotating Data — where the examples come from and how to label them, so the mirror reflects exactly what you want.