The Post-Training Pipeline: SFT → DPO → RL

Introduction

You've spent three sections learning to fine-tune a model and build the dataset for it. But supervised fine-tuning is only the first layer of how modern, capable models are actually made. This lesson opens Section 4: Advanced Post-Training with the map — the full pipeline that turns a raw base model into something helpful, aligned, and able to reason.

The big idea: building a great model is a layered stack, and each stage teaches something the previous one fundamentally can't. Pretraining gives it knowledge. SFT teaches it to follow instructions — but only by imitation. Preference tuning teaches it what people actually prefer — something you can't write into a demonstration. And RL with verifiable rewards teaches it to reason — to get verifiably-hard things right. This lesson is the bird's-eye view; the rest of Section 4 drills into each stage (DPO in L202, GRPO in L203, and more).

An infographic titled 'The Post-Training Pipeline: SFT to DPO to RL', the lesson that opens section four on advanced post-training and maps the whole stack. Building a capable, aligned model is a layered pipeline where each stage adds a capability the previous one cannot. Pretraining uses next-token prediction over internet-scale text to instill broad knowledge, producing a base model that knows a great deal but follows nothing. Supervised fine-tuning, the subject of the earlier sections, trains the model on high-quality demonstrations so it imitates them, teaching format and instruction following; SFT imitates one good answer. Preference alignment then teaches the model what people actually prefer. The key insight is that humans cannot reliably write the perfect answer or assign absolute scores, but they can reliably compare two answers and say which is better, so preference tuning learns from comparisons of a chosen versus a rejected response rather than from absolute labels. The classic three-step recipe from InstructGPT is supervised fine-tuning, then training a reward model on human preference pairs, then reinforcement learning with PPO to maximize that reward; a 1.3-billion-parameter InstructGPT was preferred over the 175-billion GPT-3. The modern 2026 default replaces the reward model and RL loop with Direct Preference Optimization, or DPO, which optimizes directly on preference pairs with a simple, stable loss. The final stage is reinforcement learning with verifiable rewards, or RLVR, using GRPO: for reasoning tasks where correctness can be automatically checked, like math and code, the model samples many answers and is rewarded for the correct ones, comparing within the group with no separate critic. This is the DeepSeek-R1 recipe. The decision framework: preference pairs point to DPO, binary thumbs to KTO, a verifiable checker to GRPO, and subjective multi-criteria feedback with a reward model and large compute to PPO. The takeaway is that SFT imitates, preference tuning optimizes for what people prefer, and RLVR teaches reasoning, applied as a stack in order.

The Four Layers of a Capable Model

Every frontier model is built in stages, applied in order — each one operating on the model the previous stage produced:

Stage	Objective	What it adds	The verb
1. Pretraining	next-token prediction on internet-scale text	broad knowledge + language	it knows
2. SFT (instruction tuning)	imitate high-quality demonstrations	instruction-following + format	it imitates
3. Preference tuning (RLHF / DPO)	optimize toward preferred responses	alignment — helpful, safe, on-tone	it optimizes
4. RL w/ verifiable rewards (RLVR / GRPO)	maximize a checkable reward	reasoning — math, code, logic	it reasons

A base model (stage 1) is a brilliant autocomplete that follows nothing. Add SFT (stage 2) and it follows instructions, but only as well as the answers you showed it. Stages 3 and 4 — the subject of this whole section — are where it learns to do things no demonstration could teach. Let's see why.

Why SFT Isn't Enough: Imitation vs. Preference

SFT has a hard ceiling, and naming it is the key to this whole section: SFT can only imitate the answers you give it. It learns "here is a good answer" — but it can't learn what makes one answer better than another, and it can't optimize for qualities you can't write down (helpfulness, tone, harmlessness, "feels right").

The breakthrough insight behind preference tuning: humans are bad at writing the perfect answer, but great at comparing two. You may not be able to author the ideal response — or assign it an absolute score of 7.4/10 — but shown two responses, you can reliably say "this one's better." That comparison is a far more scalable and reliable signal than absolute labels.

So preference tuning learns from preference pairs: a prompt, a chosen response, and a rejected one. Instead of imitating a single demonstration, the model learns the direction of human preference — and optimizes toward it. The proof is famous: InstructGPT (SFT + RLHF) at just 1.3B parameters was preferred by humans over the 175B GPT-3. Alignment, not scale, made it useful. SFT imitates; preference tuning optimizes.

The Classic Recipe: RLHF in Three Steps

The original alignment pipeline (InstructGPT, 2022) is the three-step RLHF that everything else descends from:

SFT — fine-tune the base model on demonstrations (instruction-following).
Reward model (RM) — collect preference pairs (humans rank pairs of model outputs), and train a model to predict a scalar reward: "how much would a human like this response?" The RM turns subjective preference into a number an algorithm can optimize.
RL (PPO) — use reinforcement learning to update the SFT model to maximize the reward model's score, with a KL penalty that keeps it from drifting too far from the SFT model (so it doesn't "hack" the reward into gibberish).

This works, but it's heavy: three models in play (policy, reward, reference), an unstable RL loop, and a reward model that can be gamed ("reward hacking"). The labels can come from humans (RLHF) or — increasingly — from an AI judge (RLAIF) for scale. The next idea made all of this dramatically simpler.

The Modern Stack (2026): SFT → DPO → RLVR

In 2026 the field has converged on a modular stack that swaps the heavy RL machinery for simpler, more stable methods — and adds a powerful new stage for reasoning:

SFT → instruction-following (Sections 1–3). The foundation.
Preference optimization → DPO (Direct Preference Optimization). The modern default for alignment: it skips the separate reward model and the RL loop entirely and optimizes directly on the preference pairs with a simple, classification-style loss. Same goal as RLHF, far simpler and more stable. (Variants: SimPO, IPO; and KTO for unpaired binary thumbs.) → L202.
RL with Verifiable Rewards (RLVR) → GRPO (Group Relative Policy Optimization). For reasoning, where correctness can be automatically checked (a math answer, a passing unit test). The model samples a group of answers, rewards the verifiably correct ones, and compares within the group — no reward model, no human labels, no critic. This is the DeepSeek-R1 recipe and the reason reasoning models exploded. → L203.

This is exactly the Tülu 3 recipe (SFT → DPO → RLVR) that beat the instruct versions of Llama 3.1, Qwen 2.5, and Mistral. The shift: from human-labeled rewards toward automated verification and AI feedback.

The Decision Framework: Which Method, When

You don't run every stage every time — you pick based on your goal and the data you have. The 2026 cheat sheet:

Demonstrations (input → ideal output) → SFT. Always the foundation.
Preference pairs (chosen ≻ rejected) → DPO (or SimPO / IPO). The default for style, tone, helpfulness, safety.
Binary thumbs (👍/👎, no pairs) → KTO — learns from unpaired signals.
A verifiable reward (unit test passes, math checker) → GRPO / RLVR. The biggest lever for reasoning.
Subjective, multi-criteria feedback + a trained reward model + big compute → PPO (classic RLHF) — it's online, so it explores beyond the static data, but it's expensive. Most teams pick DPO unless they can afford on-policy sampling.

And a useful axis: DPO is offline (learns from static pairs, no generation during training); PPO and GRPO are online (the model generates responses during training and is scored on them) — online explores more but costs more (the same off-policy vs on-policy idea from L198 (Model Distillation)). You'll route all of this by hand in the lab.

In Code: The Stack, End to End

Conceptually, the whole pipeline is a sequence of TRL trainers, each starting from the previous checkpoint. Here's the modern stack in outline:

# The post-training stack: each stage starts from the previous checkpoint.
from trl import SFTTrainer, DPOTrainer, GRPOTrainer

# 1. SFT - teach instruction-following by imitation (Sections 1-3).
sft = SFTTrainer(model=base, train_dataset=demos).train()        # -> model_sft

# 2. DPO - align to PREFERENCES directly from pairs (no reward model, no RL loop). L202
#    dataset rows: {prompt, chosen, rejected}
dpo = DPOTrainer(model=model_sft, train_dataset=pref_pairs).train()   # -> model_aligned

# 3. RLVR with GRPO - teach REASONING via a verifiable reward. L203
def reward(prompt, completion):       # automatic check - no human labels
    return 1.0 if math_is_correct(completion) else 0.0
grpo = GRPOTrainer(model=model_aligned, reward_funcs=[reward],
                   train_dataset=math_prompts).train()           # -> model_reasoner

# Pick the stages you need: SFT is always first; add DPO for alignment, GRPO for reasoning.

See It: The Post-Training Planner

Plan your own stack. Pick a goal (format, brand voice, helpful & safe, or reasoning), check off the data you actually have (demonstrations, preference pairs, binary thumbs, a verifiable checker), and a compute level — and watch the planner route to the right recipe, draw the pipeline with arrows, and fill the capability bars stage by stage. Try the key contrasts: switch the goal to reasoning with and without a checker (GRPO appears or doesn't); have only thumbs instead of pairs (KTO instead of DPO); go frontier compute on a helpful-&-safe goal (PPO unlocks).

The planner is the section in miniature: every recipe it draws is a path through the stages we'll build next — DPO (L202), GRPO (L203), and beyond.

Why This Matters

Knowing this map is what separates "I fine-tuned a model" from "I built an aligned, reasoning model." Most of the capability gap between a raw SFT model and a frontier assistant — the helpfulness, the safety, the refusal behavior, the step-by-step reasoning — comes from stages 3 and 4, not from more SFT. Understanding the pipeline tells you which lever to pull: if your model is unhelpful or off-tone, you need preference tuning; if it can't reason through math or code, you need RLVR; if it just won't follow the format, you need better SFT data (back to Section 3). This lesson is the table of contents for Section 4 — next we build each stage for real, starting with L202 (Direct Preference Optimization).

🧪 Try It Yourself

Route four real scenarios in the Post-Training Planner:

A coding model that must pass unit tests. Goal = reasoning, data = a verifiable checker. Which stages does it recommend, and why GRPO and not DPO?
A brand-voice chatbot. Goal = brand voice, data = preference pairs. Which method, and what does it say the difference is between this and just more SFT?
Thumbs only. Same alignment goal but you have only 👍/👎, no pairs. What changes, and why?
No preference data at all. Pick a helpful-&-safe goal with only demonstrations. Why does the planner say you can't align beyond SFT yet — and what's the fix?

Bonus: in your own words, why can a 1.3B preference-tuned model beat a 175B SFT-only one? What does preference tuning add that scale alone can't?

Mental-Model Corrections

"SFT is the whole game." SFT only teaches imitation of demonstrations. The alignment and reasoning that make models genuinely useful come from stages 3–4.
"Preference tuning needs perfect answers." The opposite — it needs comparisons (chosen ≻ rejected). Humans compare reliably even when they can't author the ideal answer.
"DPO is a kind of RL." No — DPO is offline and has no RL loop or reward model; it optimizes preference pairs directly. PPO and GRPO are the actual RL methods.
"RLHF and RLVR are the same." RLHF optimizes a learned reward model from human preferences; RLVR optimizes an automatic, verifiable reward (math/code) — no reward model, no human labels.
"You always need all four stages." You pick by goal + data: pure format → SFT may be enough; alignment → DPO; reasoning → GRPO. Run only what your goal requires.

Key Takeaways

Post-training is a layered stack, each stage adding what the last can't: Pretrain (knowledge) → SFT (instruction-following, imitates) → Preference tuning (alignment, optimizes) → RLVR (reasoning, verifies).
Why beyond SFT: SFT only imitates one good answer; preference tuning learns from comparisons (chosen ≻ rejected) — humans compare more reliably than they score. InstructGPT 1.3B beat GPT-3 175B.
Classic RLHF = 3 steps: SFT → reward model → PPO (with a KL penalty). Powerful but heavy and game-able.
Modern stack (2026): SFT → DPO (direct, no reward model/RL — L202) → RLVR/GRPO (verifiable rewards for reasoning — L203). The Tülu 3 recipe.
Decision framework: pairs → DPO, thumbs → KTO, verifiable checker → GRPO, subjective + compute → PPO. DPO is offline; PPO/GRPO are online.

Next: L202 — Direct Preference Optimization (DPO) — the modern alignment workhorse, from the math up.