Supervised Fine-Tuning (SFT): Teaching Instruction-Following

Last lesson we saw the problem: a base model is knowledge-rich but won't follow instructions, has no assistant identity, and doesn't understand chat. Post-training fixes this in two stages — and this lesson is stage one: Supervised Fine-Tuning (SFT).

The idea is beautifully simple. If the base model doesn't know how an assistant is supposed to behave, we just show it — thousands of examples of a request paired with an ideal response — and let it imitate. That's SFT in one sentence: teach the model to follow instructions by showing it good demonstrations.

You'll learn:

• The core idea: learning instruction-following by imitation
• What an SFT training example actually looks like (the chat template)
• The one mechanical trick that makes it work: masking the prompt, training on the response
• Where the data comes from, and why quality beats quantity (LIMA, ~1,000 examples)
• What SFT can't do — which sets up the next lesson, RLHF

A base model has already learned language and knowledge by imitating the internet. SFT uses the exact same skill — imitation — on much better-chosen examples. Instead of "imitate random web text," we say: "imitate this curated set of an ideal assistant answering questions."

Think of it as an apprenticeship. The base model is a brilliant graduate who has read every book but has never done the job. SFT is the few weeks of worked examples — "here's a customer request, here's how a great assistant responds" — repeated thousands of times until the new behavior sticks.

Crucially (recall last lesson): we're not adding knowledge here. We're teaching a behavior — "when you see an instruction, respond helpfully in this format" — using ability the model already has.

Every SFT example is a demonstration: a prompt and the ideal response a helpful assistant would give. To teach conversation structure, each one is wrapped in a chat template — special markers that label who's speaking (system, user, assistant):

{
  "messages": [
    { "role": "system",    "content": "You are a helpful assistant." },
    { "role": "user",      "content": "Write a haiku about the ocean." },
    { "role": "assistant", "content": "Endless blue expanse\nwaves whisper to the shoreline\nsalt air fills my lungs" }
  ]
}

Those role markers are exactly why an instruct model understands turns ("the user said X, now I, the assistant, respond") while a base model can't. A dataset is thousands of these demonstrations spanning many tasks: Q&A, summarizing, rewriting, coding, refusing unsafe requests, and more — so the behavior generalizes instead of memorizing.

Here's the one mechanical insight, and it's the heart of SFT. The training engine is identical to pretraining — predict the next token, compare to the real token, nudge the weights. We did not invent a new algorithm.

The only change is which tokens count toward the loss. We apply a loss mask: the model still reads the whole example (system + user + assistant), but the error is measured only on the assistant's response tokens. The prompt tokens are masked out — they're context, not a target.

Why? Because we don't want the model to learn to echo or invent prompts — we want it to learn to produce the response given a prompt. So we only grade it on the part it should generate.

That's the whole secret. Same loss, same optimizer, new data, plus a mask. After a pass over enough diverse demonstrations, the model has internalized: "when text is framed as a user asking me something, the likely continuation is a helpful assistant answer" — and instruction-following emerges.

Someone has to write those ideal responses. Three sources, roughly in historical order:

• Human-written (the original way). OpenAI's InstructGPT — the direct ancestor of ChatGPT — was fine-tuned on roughly 13,000 demonstrations written by human labelers. High quality, but slow and expensive.
• Synthetic / model-generated (today's workhorse). A strong existing model generates candidate answers, which humans (or another model) filter and edit. This scales demonstration-writing massively.
• Distillation. Use a top model (e.g. a frontier model) to produce the demonstrations that train a smaller one — how many capable open models bootstrap their instruct versions.

In practice modern SFT sets are a curated blend of all three, with heavy filtering for quality, diversity, and safety.

You might expect SFT to need millions of examples. It doesn't — and why tells you what SFT really does.

The LIMA result ("Less Is More for Alignment") fine-tuned a strong base model on just ~1,000 carefully curated examples and got remarkably good instruction-following. The takeaway, sometimes called the superficial alignment hypothesis: almost all the model's knowledge and ability already came from pretraining; SFT mainly teaches it which style, format, and role to surface. You're not pouring in capability — you're unlocking and directing capability that's already there.

So the priorities flip from pretraining's "more data": for SFT, a small, diverse, high-quality set beats a huge noisy one. A few rough rules of thumb practitioners use:

One bad demonstration teaches a bad habit, so curation and de-duplication matter more than raw volume.

After SFT you have an instruct model: it follows instructions, holds the assistant role, and understands chat turns. For many uses that's already enough.

But SFT has a built-in ceiling, because it only ever shows the model examples of "good." That leaves gaps:

• It can only be as good as its demonstrations — it imitates them, it can't exceed them.
• It never sees what not to do. There's no signal that response A is better than response B — only "here is one acceptable answer." It can't learn fine-grained preferences (more helpful? safer? better-toned?).
• Hard to teach nuanced trade-offs like "be helpful but refuse this category" from positive examples alone.

That missing ingredient — comparing responses and learning which humans prefer — is exactly what the next stage adds.

SFT isn't just how labs build assistants — it's the fine-tuning technique you'll most often use yourself (Container 5 goes deep). Practical takeaways you can already act on:

• Your fine-tuning data = (prompt, ideal response) pairs. If you can write/collect great example answers for your task, you can SFT a model to do it.
• Start from an instruct model, not a base one (from last lesson) — you keep its alignment and only nudge behavior.
• Obsess over data quality, not quantity. A few hundred clean, diverse, on-style examples often beat tens of thousands of sloppy ones.
• Match the chat template the model expects — wrong role markers silently wreck results.
• Parameter-efficient SFT (LoRA/QLoRA) lets you do all this on modest hardware — also coming in Container 5.

Write one SFT example. For the task 'classify a review's sentiment,' sketch a single training example in your head: a prompt (Review: "..." → Sentiment:) and the ideal response (positive).

Now the key question: during training, which part is masked out of the loss? → the prompt — the model is only graded on producing the response. That one masking trick is the whole mechanical difference between pretraining and SFT.

• "SFT is a new, complex algorithm." No — it's the same next-token training as pretraining, just on curated demonstrations with the prompt masked out of the loss.
• "SFT teaches the model new knowledge." Mostly no — it teaches behavior/format/role. The knowledge came from pretraining (recall: pretraining = knowledge, post-training = behavior).
• "More SFT data is always better." No — quality and diversity dominate; LIMA got far on ~1,000 examples. Bad examples actively hurt.
• "After SFT the model is fully aligned." Not quite — SFT only shows 'good' answers; it can't learn preferences. That's why RLHF/DPO come next.

• SFT teaches instruction-following by imitation: show the model thousands of (prompt → ideal response) demonstrations and it learns to behave like a helpful assistant.
• Each example is wrapped in a chat template (system/user/assistant) so the model learns conversation structure.
• Mechanically it's identical to pretraining — next-token prediction — with one change: a loss mask so only the response tokens are trained on, not the prompt.
• Quality ≫ quantity: curated sets of ~1,000–10,000 examples go a long way (LIMA), because SFT mostly surfaces ability pretraining already gave the model.
• SFT's limit: it only shows good answers and can't learn which response is better.

Next: we add that missing signal — RLHF, where the model learns from human preferences between responses to become more helpful, harmless, and honest.