Direct Preference Optimization (DPO)

Introduction

L201 (The Post-Training Pipeline) put DPO at the center of modern alignment: the stage where a model stops merely imitating demonstrations and starts optimizing for what people actually prefer. This lesson builds it from the ground up — and DPO is one of those rare ideas that's both theoretically beautiful and practically simple.

Classic RLHF (L201) needed three models and an unstable RL loop. DPO's insight collapses all of it into a single classification loss on preference pairs — no reward model, no reinforcement learning. The tagline from the paper says it all: "Your Language Model is Secretly a Reward Model." By the end you'll understand the implicit reward, why the dreaded partition function cancels, what the β knob really does, and the one failure mode — likelihood displacement — that has quietly un-aligned real production models.

An infographic titled 'Direct Preference Optimization, DPO', the second lesson of section four on advanced post-training and the modern workhorse for aligning a model to human preferences. DPO reframes the alignment problem from classic RLHF, which trains a separate reward model and then runs reinforcement learning, into a single supervised classification task on preference pairs. Each training example is a prompt with a chosen and a rejected response, and DPO pushes the model to make the chosen response more likely and the rejected one less likely, but always relative to a frozen reference model, which is the supervised-fine-tuned model you start from. The central insight, captured by the paper title your language model is secretly a reward model, is that the reward can be written implicitly as beta times the log ratio of the trained policy probability to the reference probability, so the policy network is simultaneously the language model and the reward model and no separate reward model is needed. The derivation works because the Bradley-Terry preference model only depends on reward differences, which makes the intractable partition function cancel out, leaving a straightforward binary cross-entropy loss. The hyperparameter beta is the leash to the reference: too low and the policy drifts far away, producing out-of-distribution gibberish and off-topic endings; too high and it barely moves, giving weak alignment; a typical default is about zero point one. DPO is offline and stable, needs only an SFT model plus a dataset of prompt-chosen-rejected triples, and is the default for most teams. The defining failure mode is likelihood displacement: because DPO optimizes only the margin between chosen and rejected, when the two responses are too similar the gradient can push both down, so the preferred response actually becomes less likely, an unintentional unalignment that has stripped refusal behavior from real models. The family of variants tunes this: IPO regularizes and caps the margin to prevent overfitting, KTO learns from unpaired binary thumbs, SimPO removes the reference model to save memory, and ORPO folds SFT and preference tuning into one stage. The takeaway is that DPO is contrastive fine-tuning that optimizes preference pairs directly, replacing the entire reward-model and RL apparatus.

The Core Idea: Contrastive Fine-Tuning

Start with the intuition, because it's simpler than the math suggests. DPO takes a preference pair — a prompt, a chosen response, and a rejected one — and does exactly one thing:

Make the chosen response more likely, and the rejected response less likely.

If that were the whole story, it would just be SFT on the chosen answers. The crucial twist is the phrase "relative to a frozen reference model." DPO keeps a frozen copy of your starting model (the SFT model — the reference π_ref) and pushes the trainable policy π_θ to make chosen more likely than the reference did, and rejected less likely than the reference did.

That "relative to reference" is what turns imitation into preference: you're not learning to copy one answer, you're learning the direction from worse → better. Think of it as contrastive SFT — SFT learns from one example; DPO learns from a contrast. Everything else is the math that makes this exact and removes the need for a reward model.

Your LM Is Secretly a Reward Model

Here's the move that won DPO its fame. The RLHF objective (L201) is: maximize a reward, but stay close to the reference (a KL penalty, strength β):

maxₚ 𝔼[ r(x,y) ] − β·KL( π_θ ‖ π_ref )

This has a known closed-form optimal policy: π_θ(y|x) ∝ π_ref(y|x)·exp( r(x,y) / β ). Rearrange it and you can write the reward in terms of the policy itself:

r̂(x, y) = β · log[ π_θ(y|x) / π_ref(y|x) ] (+ a constant)

That's the implicit reward — the reward is just β times the log-ratio of how much more likely the policy makes a response versus the reference. The policy network is the reward model. No separate model to train.

Now plug this into the Bradley-Terry preference model (the standard model: P(chosen ≻ rejected) = σ(r_chosen − r_rejected)). Because Bradley-Terry only depends on the difference of rewards, the nasty partition function Z(x) cancels — and what's left is a plain binary cross-entropy loss you can optimize with normal gradient descent.

The DPO Loss

Putting it together, the entire DPO objective is one line:

L_DPO = − log σ( β·log[π_θ(chosen)/π_ref(chosen)] − β·log[π_θ(rejected)/π_ref(rejected)] )

Read it as: take the implicit reward of the chosen minus the implicit reward of the rejected (the margin), pass it through a sigmoid, and maximize the log-likelihood that chosen ≻ rejected. Minimizing this loss:

raises the chosen response's probability relative to the reference, and
lowers the rejected response's probability relative to the reference,

by an amount scaled by β. It is a supervised classification loss — stable, offline, and trainable with the same machinery as SFT. That's the whole algorithm: no reward model, no RL rollouts, no critic. You'll watch this margin grow (and the loss fall) by hand in the lab.

β: The Leash to the Reference

β is the most important DPO hyperparameter, and it's exactly the KL-penalty strength from the RLHF objective — the leash that keeps the policy near the reference model:

β too low (loose leash) → the policy can drift far from the reference (high KL). You get out-of-distribution outputs: off-topic endings, long rambles, gibberish tokens, reward over-optimization. (A too-high learning rate does the same.)
β too high (tight leash) → the policy barely moves from the reference → weak alignment; chosen and rejected hardly separate.
A typical default is ~0.1. Tune it by watching your KL / win-rate: you want the chosen–rejected margin to grow while staying close to the reference.

This is the deep payoff of the DPO derivation: the KL constraint of RLHF is baked directly into the loss via β — you get RLHF's "stay close to the reference" guarantee for free, without running RL. Drag β in the lab and watch the drift (KL) and the margin trade off.

The Failure Mode: Likelihood Displacement

DPO's most important — and most counterintuitive — failure mode is likelihood displacement, and it follows directly from the loss optimizing the margin (chosen − rejected), not the chosen probability itself.

Here's the trap: a gradient step can lower the rejected probability faster than it raises the chosen one — so the margin grows (the loss happily drops) even while the chosen response's absolute probability falls. When chosen and rejected are too similar (they share tokens / structure), the gradient that suppresses rejected drags the chosen down with it. Probability mass leaks away from the response you wanted.

This isn't theoretical. Standard DPO has been measured dropping refusal rates of aligned models — Llama-3-8B-Instruct from 74.4% → 33.4%, Gemma-2B-IT from 80.5% → 54.8% — unintentionally un-aligning them by displacing probability away from preferred refusals. The fixes: use distinct chosen/rejected pairs (not near-identical), DPO-Positive (DPOP) which adds a term to keep the chosen's absolute likelihood up, or a reference-free variant. You can trigger this live in the lab with the "subtle" data toggle and watch the chosen bar drop.

The DPO Family: IPO, KTO, SimPO, ORPO

DPO spawned a family of variants, each fixing a specific limitation — and knowing the map tells you which to reach for:

IPO (Identity PO) — DPO can overfit the preference pairs (the margin grows unboundedly while win-rate stalls). IPO adds a regularizer that caps the margin. Reach for it when training margin saturates but validation win-rate doesn't improve.
KTO (Kahneman-Tversky) — for unpaired binary feedback (👍/👎 per response, no matched pairs). Uses prospect-theory weighting (losses hurt more than gains). Reach for it when all you have is thumbs.
SimPO — reference-free: drops the frozen reference model entirely (uses the average log-prob + a target margin), so no second model in memory. Often beats DPO on AlpacaEval / Arena-Hard.
ORPO — folds SFT and preference tuning into one stage (an odds-ratio term), eliminating the SFT→DPO distribution shift and a whole training pass.

The throughline: they're all "learn from comparisons" with different reference handling, regularization, and data shapes. DPO is the default; the variants are targeted fixes.

In Code: DPO with TRL

In practice DPO is a few lines — start from your SFT checkpoint, feed {prompt, chosen, rejected} triples, set β, and TRL handles the frozen reference model for you:

from trl import DPOTrainer, DPOConfig

# Dataset: one row per preference pair.
#   {'prompt': 'How do I reset my password?',
#    'chosen':  'Click "Forgot password", check your email for the reset link...',
#    'rejected':'Just google it, it is not that hard.'}
#   -> use DISTINCT chosen/rejected to avoid likelihood displacement.

trainer = DPOTrainer(
    model=sft_model,          # START from the SFT checkpoint (this is also the reference)
    train_dataset=pref_pairs, # thousands to tens of thousands of triples
    eval_dataset=val_pairs,
    args=DPOConfig(
        beta=0.1,             # the KL leash to the reference (~0.1 default)
        learning_rate=5e-6,   # SMALL - DPO is sensitive; too high = drift/gibberish
        loss_type='sigmoid',  # 'sigmoid'=DPO · 'ipo'=IPO · 'kto_pair' etc.
    ),
)
trainer.train()
# Watch: rewards/margins should RISE, rewards/chosen should stay >= 0 (else displacement),
# and KL from the reference should stay modest.

See It: The DPO Lab

Drive DPO by hand. Drag training progress and watch the chosen bar rise above the frozen reference while rejected falls — the margin grows and the loss drops. Now probe the knobs: drop β to ~0.02 and push training up to see the drift (KL explodes → gibberish); raise β to see alignment barely move. Then flip the data to "subtle" and watch likelihood displacement — the chosen bar drops even as the margin grows. Finally switch to SimPO (the reference disappears) and IPO (the margin caps).

Interactive: the DPO Lab. On a concrete preference pair for the prompt about resetting a password, the user drags a training-progress slider and a beta leash, picks a variant of DPO, SimPO, or IPO, and toggles the data between distinct and subtle near-identical responses, then watches the mechanism unfold. Two probability bars, one for the chosen response and one for the rejected response, each show the frozen reference next to the moving policy, so the user sees the policy make the chosen response more likely and the rejected one less likely relative to the reference. Live metrics show the implicit-reward margin, which is beta times the difference of log ratios, the DPO loss as negative log sigmoid of the margin, and the KL divergence from the reference, a measure of drift. A verdict names the state: untrained when the policy equals the reference, healthy DPO when chosen rises and rejected falls near the reference, drift when beta is too low and KL explodes into gibberish, weak alignment when beta is too high, and the famous likelihood displacement when the responses are too similar and the chosen response's probability falls in absolute terms even as the margin grows, which is unintentional unalignment. SimPO removes the reference bars to show it is reference-free, and IPO caps the margin to show its regularization.

The lab makes the whole algorithm visible: DPO is just push chosen up, rejected down, relative to the reference — with β as the leash and displacement as the trap.

Why This Matters

DPO is the default alignment tool of 2026 — it's how most teams take an SFT model and make it helpful, safe, and on-brand without standing up a reward model and an RL cluster. Understanding it from the implicit reward up means you can debug a run that's gone wrong (margin growing but quality dropping? likelihood displacement — check your pairs; gibberish? β too low), choose the right variant (thumbs → KTO, memory-tight → SimPO, overfitting → IPO), and reason about the quality of your preference data — which, as always, is the real lever. It's the workhorse half of Section 4; next we cross into the other half — reinforcement learning for reasoning — with L203 (Reasoning Fine-Tuning with GRPO).

🧪 Try It Yourself

Probe DPO's mechanism and its failure in the DPO Lab:

Healthy run. Distinct data, β≈0.1, drag training to ~70%. What happens to the chosen bar, the rejected bar, and the margin — and what does "relative to the reference" mean here?
Cause displacement. Switch to subtle data and train. Why does the chosen probability fall even though the loss is dropping? (Say it in terms of margin vs absolute probability.)
β extremes. Set β to 0.02 then 0.5 at high training. Describe the two failure modes (drift vs weak alignment) and what KL does in each.

Bonus: DPO needs no reward model. In one sentence, why — what cancels in the Bradley-Terry derivation, and what plays the role of the reward?

Mental-Model Corrections

"DPO is reinforcement learning." No — DPO is offline supervised classification on pairs. There's no RL loop, no rollouts, no reward model. (PPO/GRPO are the RL methods.)
"DPO just trains on the chosen answers." That's SFT. DPO trains on the contrast (chosen vs rejected) relative to a frozen reference — that's what encodes preference.
"Lower β is always better alignment." No — too-low β removes the leash → drift and gibberish. β is a regularizer (~0.1); tune it.
"If the loss drops, the model improved." Not necessarily — likelihood displacement lets the margin (and loss) improve while the chosen response gets less likely. Watch the absolute chosen reward, not just the margin.
"DPO replaces the reward model with nothing." It replaces it with an implicit reward — β·log(π_θ/π_ref) — so the policy is the reward model.

Key Takeaways

DPO = contrastive fine-tuning: push chosen ↑ / rejected ↓ relative to a frozen reference (the SFT model). No reward model, no RL — a single classification loss on {prompt, chosen, rejected}.
Your LM is secretly a reward model: the implicit reward is r̂ = β·log(π_θ/π_ref); Bradley-Terry only needs reward differences, so the partition function cancels.
β is the leash (the RLHF KL penalty, baked in): too low → drift/gibberish, too high → weak alignment; ~0.1 default.
Likelihood displacement is the trap: optimizing only the margin can make the chosen answer less likely too (it has un-aligned real models). Use distinct pairs / DPOP.
The family: IPO (cap the margin), KTO (unpaired thumbs), SimPO (reference-free), ORPO (one stage). DPO is the default.

Next: L203 — Reasoning Fine-Tuning with GRPO — crossing from preference tuning into reinforcement learning with verifiable rewards.