Model Distillation

Introduction

L197 (Synthetic Data Generation) ended on a distinction: generating data with a teacher is one thing; compressing a model's capability into a smaller one is another. That second thing is model distillation — and it's how the frontier reaches your phone, your edge device, and your budget.

The promise is almost too good: take a giant, expensive teacher and train a small, cheap student to imitate it — keeping ~95% of the quality at 5–30× lower cost and ~4× faster inference. That's how a 671B reasoning model becomes a 7B you can actually deploy. This lesson covers the mechanism (soft targets, dark knowledge, temperature), the taxonomy (white-box vs black-box, off-policy vs on-policy), the headline use case (reasoning distillation), and the one hard limit you can never engineer around.

An infographic titled 'Model Distillation', the fourth lesson of section three on dataset engineering, on the teacher-student technique that compresses a big model's capability into a small, cheap one. Distillation trains a compact student to imitate a high-capacity teacher by matching its predictive distribution rather than just the hard label. The key idea is dark knowledge: the teacher's soft targets, the full probability spread over tokens such as cat seventy percent, kitten twenty percent, animal ten percent, carry similarity and uncertainty information far richer than a one-hot answer, and a temperature parameter softens the softmax to expose it. The student is trained to match the teacher's softened distribution by minimizing the Kullback-Leibler divergence between them. The payoff is large: five to thirty times lower cost, about four times faster inference, and ninety-five to ninety-seven percent of the teacher's performance, which is what makes edge and on-device deployment feasible; DistilBERT is forty percent smaller and sixty percent faster while keeping ninety-seven percent of BERT. There are two access regimes. White-box distillation has the teacher's logits and internals, enabling true logit and feature matching. Black-box distillation only has the teacher's text outputs through an API, so it distills on generated sequences, which is exactly the synthetic data pipeline from the previous lesson; this is why DeepSeek-R1 distillation was simply supervised fine-tuning of small students on eight hundred thousand teacher-generated reasoning traces. There are also two policy regimes. Off-policy distillation trains the student on the teacher's static outputs and is simple but suffers exposure bias, because the student never learns to recover from its own mistakes. On-policy distillation, the 2026 frontier, has the student generate its own rollouts and uses the teacher's per-token log-probabilities as dense feedback on the states the student actually visits. The headline use case is reasoning distillation, transferring long chain-of-thought into small models. The hard limit is that the student can never exceed the teacher. The takeaway is that distillation is capability compression by imitation of soft targets, giving near-teacher quality at a fraction of the cost.

The Mechanism: Soft Targets & Dark Knowledge

Ordinary fine-tuning teaches with hard labels: the answer is cat, full stop. Distillation does something cleverer — it teaches with the teacher's full probability distribution.

Ask a teacher to predict the next token for "the small animal purring on my lap is a ___" and it might say cat 70%, kitten 20%, tiger-cub 7%, puppy 2%, hamster 1%. A hard label throws all of that away and keeps only cat. But the distribution itself is gold: it encodes that cat ≈ kitten ≈ tiger-cub are all feline, that puppy is close-ish, that hamster is unlikely. This is the teacher's "dark knowledge" — the similarity and uncertainty structure that a one-hot label can't express.

To expose it, distillation raises the softmax temperature (T) (Hinton et al., 2015). Higher T softens the distribution, lifting the runner-up tokens into view; lower T sharpens it back toward the hard label. The student is then trained to match the teacher's softened distribution — minimizing the KL divergence between them — so it learns not just the answer but the whole landscape around it. You'll drag this temperature yourself in the lab and watch the dark knowledge appear.

Why Distill: Cheaper, Faster, Deployable

Distillation is a capability-compression move, and the numbers are why it's everywhere in 2026:

5–30× lower cost and ~4× faster inference, while retaining 95–97% of the teacher's performance.
Edge & on-device becomes feasible — a distilled student small enough to run on a phone or in a low-latency service.
The canonical example: DistilBERT — 40% smaller, 60% faster, 97% of BERT's performance, 44M fewer parameters.

When do you reach for it versus the earlier tools?

Fine-tuning (L192–L194) — when accuracy on a niche task is paramount and you have labels.
Distillation — when you need speed, cost, and scale: the same capability in a model you can afford to serve to millions or run at the edge.

In practice they blend: the most common recipe is distillation-as-fine-tuning — generate teacher outputs (L197), then SFT a small student on them. Which brings us to how the knowledge actually transfers.

White-Box vs Black-Box: Do You Have the Logits?

The first fork is access — can you see inside the teacher?

White-box distillation — you have the teacher's logits / internal features (an open-weight teacher like Llama or Qwen, or your own model). You can do true logit matching (response-based) and even align intermediate features (feature-based, e.g. MiniLM). This is the richest signal — the full soft distribution, every token.
Black-box distillation — you only have the teacher's text outputs via an API (GPT-5, Gemini, Claude). You can't see logits, so you distill on the generated sequences — sequence-level KD. Methods like Lion operate here.

Here's the key connection: black-box distillation IS the L197 pipeline. Generating a teacher's text answers and SFT-ing a student on them is sequence-level distillation. That's exactly why DeepSeek-R1's distillation was "just" SFT on 800K teacher-generated reasoning traces — no logits, no teacher-student loop, pure black-box. (And recall the L197 legal note: black-box distillation off a frontier API is bound by its terms of service — use open-weight teachers or an official distillation API.)

Also worth knowing: knowledge is categorized as response-based (mimic outputs — most common), feature-based (align internal activations), and relation-based (mimic relationships between examples).

Off-Policy vs On-Policy: Whose Mistakes?

The second fork is policy — whose outputs does the student train on?

Off-policy (the default; this is SFT) — the student learns from the teacher's static, pre-generated outputs. Simple and massively scalable. But it suffers exposure bias: the student only ever sees the teacher's perfect trajectories, and never learns to recover from its own mistakes — because at training time it never makes any. DeepSeek-R1-Distill is strictly off-policy (the 800K-trace SFT).
On-policy (the 2026 frontier) — the student generates its own rollouts, and the teacher grades them per-token (its log-probabilities act as a dense reward) on the states the student actually visits. This directly fixes exposure bias. Thinking Machines Lab showed it replicates the Qwen3 recipe at a fraction of RL compute — dense, on-policy supervision is both effective and efficient.

The trade-off: off-policy is simpler and cheaper to run (just generate + SFT); on-policy is more sample-efficient and avoids exposure bias but needs the student-in-the-loop machinery. Start off-policy; reach for on-policy when the student plateaus or you're distilling hard, multi-step behavior.

In Code: Black-Box (Sequence-Level) Distillation

The most common distillation you'll run is black-box and off-policy — and it looks exactly like the synthetic-data → SFT pipeline, because that's what it is:

# BLACK-BOX, OFF-POLICY distillation = generate teacher traces -> SFT the student.
# (This is how DeepSeek-R1-Distill was made: 800K traces -> plain SFT.)
traces = []
for prompt in prompts:
    out = teacher(prompt, temperature=0.7)     # the teacher's full answer / reasoning chain
    if judge(out).score >= 4:                  # filter - an unfiltered set is worse (L197)
        traces.append({'messages': [{'role': 'user', 'content': prompt},
                                    {'role': 'assistant', 'content': out}]})

# Train the small STUDENT to imitate the teacher's text (sequence-level KD).
from trl import SFTTrainer, SFTConfig
SFTTrainer(model=student_8B, train_dataset=traces,
           args=SFTConfig(learning_rate=2e-4, num_train_epochs=3)).train()
# Result: a 7-8B student that inherits the teacher's behavior at ~5-30x lower cost.

In Code: White-Box (Logit) Distillation

When you own the teacher's logits, you can transfer the full soft distribution — the dark knowledge — via a temperature-softened KL loss. This is the classic Hinton recipe:

import torch.nn.functional as F

# WHITE-BOX: match the teacher's SOFTENED distribution (the dark knowledge), not just
# the hard label. T raises the temperature to expose the runner-up tokens.
def distill_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.5):
    soft_teacher = F.softmax(teacher_logits / T, dim=-1)         # soft targets
    soft_student = F.log_softmax(student_logits / T, dim=-1)
    kd  = F.kl_div(soft_student, soft_teacher, reduction='batchmean') * (T * T)
    ce  = F.cross_entropy(student_logits, labels)                # the usual hard-label loss
    return alpha * kd + (1 - alpha) * ce      # blend: learn the distribution AND the answer

# The (T*T) term keeps gradient magnitudes stable as you raise the temperature.
# Higher alpha -> trust the teacher's soft targets more than the hard labels.

See It: The Distillation Lab

Feel the mechanism, then plan a distillation. In Part 1, drag the temperature and watch the teacher's token bars soften — at T≈1 it's basically a hard label (cat), but crank it up and the dark knowledge appears (cat ≈ kitten ≈ tiger-cub). Then in Part 2, configure a real run: pick a teacher and student size, choose white-box vs black-box, off- vs on-policy, and response vs reasoning — and read the cost / latency / quality trade-off plus the verdict.

Interactive: the Distillation Lab, in two parts. Part one makes the mechanism tangible: the user drags a temperature slider on a teacher's next-token distribution for the prompt about a small animal purring on a lap, and the soft-target bars soften as temperature rises, revealing the dark knowledge that cat, kitten, and tiger-cub are all feline while puppy and hamster are less likely, with an entropy readout showing the extra signal a hard label throws away. Part two is a configurator for a real distillation: the user picks the teacher size, the student size, the access regime of white-box logits versus black-box API text, the policy of off-policy supervised fine-tuning versus on-policy student rollouts, and whether to distill responses or reasoning chains. Live readouts show predicted cost reduction, latency reduction, and quality retained, and a binding-issue verdict names the dominant problem, whether a capability gap from too small a student, exposure bias from off-policy training, the sequence-level nature of black-box distillation, reasoning needing capacity, or a strong plan. An arrow pipeline shows teacher to outputs to training to a small edge-deployable student.

Try to break it: distill a 671B teacher into a 1B student and watch the capability gap crater quality; pick reasoning into a tiny student and get the "reasoning needs capacity" warning. The student never exceeds the teacher — distillation compresses, it doesn't create.

Why This Matters

Distillation is what makes frontier capability affordable and deployable — it's behind nearly every small model that punches above its weight (the entire R1-Distill family, DistilBERT, and every "GPT-4o-mini fine-tuned on GPT-4o outputs" pipeline). For an AI engineer it's the lever you pull when the model works but is too slow or too expensive to ship: same behavior, a fraction of the cost. It also ties this whole section together — black-box distillation = the L197 synthetic-data pipeline with a compression goal — and it sets up the final dataset skills: cleaning what you've generated (L199 — Cleaning, Dedup & Formatting) and assembling it all (L200 — Hands-On: Build a Fine-Tuning Dataset).

🧪 Try It Yourself

Explore the trade-offs in the Distillation Lab:

Dark knowledge. Set temperature to 1.0, then to 4.0. What happens to the bars and the entropy, and what extra thing does the student learn at high T that a hard label can't teach?
The capability gap. Distill 671B → 1B. What happens to quality kept, and why can't you fix it with more data? (State the hard limit in one sentence.)
Exposure bias. Compare off-policy vs on-policy for a reasoning student. What problem does on-policy fix, and how (whose mistakes does it train on)?

Bonus: you only have API access to the teacher (no logits). Which distillation type are you forced into, and which earlier lesson's pipeline does it turn out to be?

Mental-Model Corrections

"A distilled student can surpass its teacher." No — distillation copies a capability; the student can't exceed the teacher. It compresses, it doesn't create.
"Distillation just means training on the teacher's answers." That's black-box distillation. White-box transfers the full soft distribution (logits) — the dark knowledge — which a single answer can't carry.
"Distillation and synthetic data are different things." Black-box distillation IS the synthetic-data pipeline (L197) — generate teacher text, SFT the student. The difference is the goal: compression.
"Off-policy SFT is all there is." It has exposure bias (the student never sees its own mistakes). On-policy distillation fixes it — the 2026 frontier.
"Any student size works." Too small a student can't absorb a huge teacher (capability gap), and reasoning distillation needs real capacity (R1-Distill starts ~7B).

Key Takeaways

Distillation = capability compression by imitation: a small student matches a big teacher's soft targets (the dark knowledge), exposed by temperature and matched via KL divergence.
The payoff: 5–30× cheaper, ~4× faster, 95–97% retained — and edge-deployable (DistilBERT: 40% smaller, 60% faster, 97%).
Access: white-box (logits → full distribution) vs black-box (API text → sequence-level = the L197 pipeline; mind the ToS).
Policy: off-policy (SFT on traces — simple, exposure bias) vs on-policy (student rollouts graded by the teacher — the 2026 frontier).
Headline use: reasoning distillation (DeepSeek-R1 → 800K traces → small students). Hard limit: the student never exceeds the teacher.

Next: L199 — Cleaning, Dedup & Formatting — the cleanup pass that turns all this raw, sourced, synthetic, and distilled data into a training-ready set.