Sourcing & Annotating Data

Introduction

L195 (Data Quality, Coverage & Quantity) told you what a good dataset is. This lesson is the how: the two hands-on jobs that actually produce one — sourcing (where do the examples come from?) and annotating (how do you label them so the model learns the right thing?).

These map cleanly onto the L195 axes: sourcing buys you coverage (does the data match production?), and annotation buys you quality (are the labels correct and consistent?). Get these two right and the hyperparameters from L194 (Fine-Tuning Hyperparameters & Tactics) barely matter; get them wrong and no knob can save you. By the end you'll know the five sources and their trade-offs, the annotation workflow that makes labels repeatable, the model-assisted pattern that 2026 teams actually use, and the legal layer you skip at your peril.

An infographic titled 'Sourcing & Annotating Data', the second lesson of section three on dataset engineering, which answers the two practical questions behind every fine-tuning set: where do the examples come from, and how do you label them well. Sourcing decides coverage, the match to the production distribution, and annotating decides quality, whether the labels are correct and consistent. There are five sources, each with trade-offs. Production logs and internal data, such as support tickets, transcripts, and historical documents, are the gold source because they already match the production distribution, but they need cleaning, de-identification of personal data, and often a better target answer. Expert-written examples give the highest quality and full control of coverage and edge cases but are slow and expensive, and are used for a seed set of roughly five hundred to a thousand. Public datasets such as Alpaca, FLAN, and Dolly are fast and large but carry licensing traps, since Alpaca is non-commercial, and may not match your distribution. Synthetic data, covered next lesson, scales cheaply from a seed but inherits the teacher's ceiling. User feedback and preference data feed later preference tuning. The recommended blend is an expert seed plus production logs plus targeted synthetic to fill gaps. Annotating is where the model's mirror is set, so consistency is the whole game and it is measured by inter-annotator agreement, Cohen's kappa for two annotators or Fleiss' kappa for more, where a kappa below roughly point six means the guideline is ambiguous, not that the annotators are bad. The workflow is a written guideline with a rubric and a label taxonomy, calibration sessions on the edge cases, a gold standard set, and a measured agreement gate that loops back to revise the guideline when agreement is low. The dominant 2026 pattern is model-assisted annotation, where an LLM pre-labels a first pass and humans review and correct, with active learning surfacing the uncertain examples, cutting human work by about half, though humans stay in the loop because models are overconfident and biased. A legal layer runs underneath: redact personal data before training, respect dataset licenses and copyright, and document a lawful basis. The takeaway is that sourcing buys coverage and annotation buys quality, and a clear guideline measured by agreement is the highest-leverage move.

Two Jobs: Sourcing = Coverage, Annotating = Quality

Building a dataset is two distinct skills, and confusing them is why teams flail:

Sourcing is gathering the raw inputs — the prompts/questions/tickets the model must handle. This is where coverage is won or lost: if your raw inputs don't span the real production distribution (including edge cases), the model never sees them. Sourcing is a distribution problem.
Annotating is attaching the right output to each input — the ideal response or label. This is where quality is won or lost: the model imitates your labels exactly (the L195 mirror), so a wrong or inconsistent label is a wrong lesson. Annotation is a consistency problem.

You need both. The richest, most representative raw inputs are worthless with sloppy labels; impeccable labels on a narrow, unrepresentative sample give you a model that's precise about the wrong things. Let's take each job in turn.

Where Data Comes From: Five Sources

There are five places fine-tuning examples come from, and the art is blending them. Each trades off coverage, quality, quantity, cost, and legal risk:

Source	Strength	Watch out for
Production logs / internal data (tickets, transcripts, docs)	Gold for coverage — it already is your production distribution; cheap & real	Needs cleaning + PII redaction; logs show what happened, not the ideal answer
Expert-written from scratch	Highest quality; total control of coverage & edge cases	Slow, expensive — used for the seed (~500–1,000)
Public datasets (FLAN, Dolly, OpenAssistant, Alpaca…)	Free, fast, large	Licensing (Alpaca is non-commercial); generic, may not match you; variable quality
Synthetic (teacher model)	Cheap scale + coverage control	Quality ceiling = the teacher; can amplify bias — L197 (Synthetic Data Generation)
User feedback / preference data (👍/👎, edits, rankings)	Real signal of what users want	Sparse, noisy — feeds preference tuning (DPO, Section 4)

Notice these line up with the L195 axes: logs win on coverage, experts win on quality, synthetic wins on quantity. No single source is best — the strategy is to combine them.

A Sourcing Strategy That Works

The 2026 default that strong teams converge on:

Start with an expert seed — ~500–1,000 carefully hand-written examples that define the task and deliberately cover the edge cases. This anchors quality and coverage from day one (LIMA-style: a small impeccable set beats a large sloppy one).
Mine production logs for real inputs — they bring the true distribution and the long tail you'd never think to write. (De-identify first — see the legal section.)
Fill gaps with targeted synthetic — for under-covered categories, expand the seed with a teacher model, then filter the bottom 10–20% with a judge (L197).

The mindset shift for 2026: the scaling era of "scrape everything" is over. Winning teams build curated, expert-reviewed sets and obsess over annotation precision — "a precise label is an expert label." Volume matters far less than representative coverage + consistent labels, which is exactly the next job.

Annotating: Consistency, Measured

Because the model imitates your labels, the deadliest annotation failure isn't being wrong — it's being inconsistent. If annotator A labels a ticket Billing and annotator B labels the same ticket Account, you've taught the model contradictory things, and it learns the noise.

So you measure consistency with Inter-Annotator Agreement (IAA) — the single most important metric in annotation. The standard is Cohen's κ (kappa) for two annotators (Fleiss' κ for more), which corrects raw agreement for chance:

κ ≥ 0.8 — strong agreement; the labels are repeatable. Scale the team.
κ 0.6–0.8 — borderline; tighten the edge-case rules.
κ < 0.6 — the guideline is ambiguous.

That last line is the insight most people miss: low agreement is not a sign of bad annotators — it's a sign of an unclear guideline. When κ is low, the fix is to clarify the instructions (and run a calibration session), not to add more QA or hire "better" people. You'll feel this directly in the lab: flip the guideline from vague to clear and watch κ jump on the very same tickets.

The Annotation Workflow

Consistent labels come from a process, not from hoping. The loop:

Write a guideline + rubric — a label taxonomy, crisp definitions, and (most important) explicit rules for the edge cases.
Calibrate — annotators label a shared batch together, surface disagreements, and align on the hard cases before scaling.
Build a gold set — a small expert-labeled "ground truth" to score every annotator against.
Measure IAA — compute κ. If it's low, revise the guideline (loop back to step 1).
Scale only once agreement is high.

The guideline is the real product here — it's the artifact that makes labels repeatable across a team and across time. Treat it like code:

# The annotation GUIDELINE is the real product - it makes labels consistent.
GUIDELINE = {
    'labels': ['Billing', 'Account', 'Technical'],
    'definitions': {
        'Billing':   'money, charges, refunds, invoices',
        'Account':   'identity, login credentials, profile, access',
        'Technical': 'the app misbehaving - bugs, crashes, errors',
    },
    # Edge-case RULES are where agreement is won or lost - write them down.
    'edge_rules': [
        'Login broken by an app update -> Technical (NOT Account)',
        'Downloading an invoice        -> Billing (even if it lives in settings)',
        'Ambiguous / multiple intents  -> label the PRIMARY intent',
    ],
    # Gold examples calibrate every annotator to the same bar.
    'gold': [
        {'text': 'I was charged twice',           'label': 'Billing'},
        {'text': "Can't log in after the update", 'label': 'Technical'},
    ],
}

Model-Assisted Annotation: Pre-label, Then Review

Hand-labeling everything is slow, so the dominant 2026 workflow is a hybrid: an LLM pre-labels a first pass (zero/few-shot, given your guideline) and humans review and correct. Two multipliers make it efficient:

Pre-labeling turns annotation into the much faster task of accepting or fixing a suggestion — cutting human effort by ~50%+. Models fine-tuned on well-reviewed LLM labels can rival those trained on fully human-labeled data.
Active learning spends human attention where it matters: surface the uncertain / novel examples (low model confidence) for review, and auto-accept the confident easy ones.

The non-negotiable caveat: the LLM is not the sole source of truth. Models are overconfident and inherit biases, so a human stays in the loop — especially for safety-critical or domain-specific labels. Then you gate the dataset on agreement before training:

# Dominant 2026 workflow: LLM PRE-LABELS, humans REVIEW - then gate on agreement.
def pre_label(ticket):                    # zero/few-shot LLM first pass
    return llm(GUIDELINE, ticket)         # -> {label, confidence}

def needs_human(s):                       # ACTIVE LEARNING: only the uncertain ones
    return s.confidence < 0.85            # auto-accept the confident easy cases

reviewed = []
for t in tickets:
    s = pre_label(t)
    label = human_correct(s) if needs_human(s) else s.label   # human stays in the loop
    reviewed.append(label)

# GATE the dataset on inter-annotator agreement BEFORE you train.
kappa = cohen_kappa(reviewed, second_annotator(tickets))
assert kappa >= 0.8, 'kappa < 0.8 -> guideline is ambiguous; revise it, do not just add QA'

The Legal Layer: PII, Licensing & Consent

Skip this and you can poison a model or expose your company to real liability. Three checks run underneath all sourcing:

PII / privacy. Production logs hide personal data in free text — a name in a clinical note, an address in a call transcript — not just in tidy labeled fields a regex can catch. Train on it and that PII can surface in unrelated outputs, triggering GDPR exposure (enforcement passed €7.1B in fines) and right-to-be-forgotten obligations (which can force costly unlearning/retraining). De-identify before training.
Licensing / copyright. Public datasets carry licenses — Alpaca is non-commercial, and training on copyrighted data is presumptively infringing absent a defense. Check provenance (the MIT Data Provenance Explorer traces dataset lineage), and note your fine-tuned weights' license is constrained by the base model's license too.
Consent / lawful basis. Under GDPR you need a documented lawful basis for personal data, a DPIA for high-risk processing, and proof the data was lawfully obtained.

The habit: know the provenance and rights of every example before it enters the set.

See It: The Annotation Lab

Become the annotator. Label the five tickets, then read your Cohen's κ against Annotator B. Now run the key experiment: with the vague guideline, notice how the borderline tickets split you apart — then hit clarify the rubric and watch κ jump on the exact same tickets. That's the whole lesson. Then turn on LLM pre-label (review the suggestions — one is wrong!) and active learning to feel the 2026 throughput multipliers, and watch the IAA gate in the pipeline open only when agreement is high.

Interactive: the Annotation Lab. The user becomes the annotator and labels five real support tickets by clicking one of three labels on each card, while a live Cohen's-kappa-style score compares their labels to a second annotator. The centerpiece is a guideline toggle: with a vague guideline the borderline tickets are genuinely ambiguous and agreement is low, but switching to a clear rubric, which spells out the edge-case rules, makes the same person on the same tickets agree with the second annotator and the kappa jumps, proving that low agreement means an ambiguous guideline rather than bad annotators. Two further toggles model the dominant 2026 workflow: LLM pre-label pre-fills a suggested label to review and correct and reports the throughput gain and roughly half the human effort saved, and active learning surfaces the uncertain tickets first so human attention goes where the model is unsure. A pipeline strip with arrow connectors shows raw data to pre-label to human review to an agreement gate to dataset, with a dashed loop back to revise the guideline whenever kappa is low. A verdict reads the kappa and names the fix.

The takeaway you can feel: you didn't become a better annotator between the two runs — the guideline got clearer. Consistency is an instructions problem.

Why This Matters

Sourcing and annotation are the unglamorous 80% of a fine-tuning project — and where it's actually won. The engineer who mines production logs for coverage, writes a sharp guideline, measures κ, and leans on LLM pre-labeling ships a clean, representative dataset in days; the one who scrapes a public set and hand-waves the labels ships a model that's inconsistent, narrow, and possibly a legal liability. This is also the connective tissue of the section: L197 (Synthetic Data Generation) and L198 (Model Distillation) are sourcing at scale, L199 (Cleaning, Dedup & Formatting) is the cleanup pass, and L200 (Hands-On: Build a Fine-Tuning Dataset) puts it all together.

🧪 Try It Yourself

Run the guideline experiment, then answer:

In the Annotation Lab, label all five tickets under the vague guideline and note your κ. Now switch to the clear rubric (re-label if needed) and note κ again. Why did it change when you didn't?
Turn on LLM pre-label. One suggestion is wrong — find it. What does this prove about using an LLM as the sole labeler?
For a support-ticket model, rank the five sources by what you'd use first and say why — which one gives you coverage, and which gives you quality?

Bonus: your production logs contain customer names and card numbers. List two things you must do before those logs go into a training set.

Mental-Model Corrections

"Low annotator agreement means I need better annotators." Almost always it means the guideline is ambiguous. Fix the instructions and calibrate — don't blame the people.
"More annotators = better labels." Not without a clear rubric and measured κ — more people just means more inconsistent labels, which the model dutifully imitates.
"Public datasets are free to use." Free to download ≠ free to use. Check the license (Alpaca is non-commercial) and copyright/provenance.
"Let the LLM label everything." It's a great first pass, but it's overconfident and biased — humans must review, especially the uncertain cases.
"PII is just the fields I can filter." It hides in free text (notes, transcripts). De-identify the body, not just the columns, before training.

Key Takeaways

Two jobs: sourcing buys coverage (match the production distribution), annotation buys quality (correct, consistent labels). You need both.
Five sources, blended: production logs (gold for coverage), an expert seed (~500–1,000) for quality, public datasets (mind the license), synthetic for scale (L197), preference data for later tuning.
Consistency is measured: Inter-Annotator Agreement (Cohen's/Fleiss' κ). κ < 0.6 ⇒ the guideline is ambiguous, not the annotators — clarify and calibrate.
The workflow: guideline+rubric → calibrate → gold set → measure κ → revise → scale. The guideline is the product.
Model-assisted is the 2026 default: LLM pre-labels, humans review (+ active learning) for ~50% less effort — but humans stay in the loop.
Mind the law: redact PII in free text, respect licenses/copyright, document a lawful basis.

Next: L197 — Synthetic Data Generation — how to turn a small seed into a large, diverse, filtered set with a teacher model.