RLHF: Aligning with Human Preferences

SFT got us most of the way: the model now follows instructions and acts like an assistant. But we ended last lesson on its ceiling — SFT only ever shows the model examples of "good." It never learns that response A is more helpful than response B, or that one answer is safer or better-toned than another. It imitates; it doesn't judge.

RLHF — Reinforcement Learning from Human Feedback — adds exactly that missing signal. Instead of "here is one ideal answer," we now ask humans a much easier question: "which of these two answers is better?" — and train the model to produce more of what people prefer. This single idea, more than any other, is why ChatGPT and Claude feel genuinely helpful rather than merely competent.

You'll learn:

• Why comparing answers beats demonstrating them
• The three steps of RLHF: SFT → reward model → RL (PPO)
• What a reward model is and how the KL leash keeps RL from going off the rails
• Reward hacking, and the modern shift to RLAIF / Constitutional AI
• A preview of DPO — the simpler successor (next lesson)

The whole leap of RLHF is changing the kind of feedback we give the model.

• SFT feedback: "Here is the ideal response. Imitate it." (one example of good)
• RLHF feedback: "Here are two responses. This one is better." (a comparison)

Why is comparison so much more powerful? Three reasons:

1. Ranking is easier than authoring. Writing the perfect answer to "explain quantum tunneling to a 10-year-old" is hard; glancing at two attempts and picking the better one is easy. So humans can give far more feedback, far more cheaply.
2. It captures relative quality — the subtle stuff SFT can't express: more helpful, more honest, warmer tone, better-structured, safer. "Good vs. better" is a richer signal than "here's one good one."
3. It teaches the model what not to do. Every comparison has a loser. For the first time the model gets a signal about worse responses, not just good ones.

That's the heart of it. Now we need machinery to turn a pile of "A is better than B" judgments into a model update.

Classic RLHF — the recipe behind InstructGPT and the first ChatGPT — has three steps. Step 1 is the SFT we just did; the two new ideas are a reward model and an RL update:

1. Start from the SFT model. Our instruction-following assistant is the starting point.
2. Train a reward model (RM). Collect human preferences (A vs. B) and train a separate model that learns to score any response by how much a human would like it.
3. Optimize with RL (PPO). Let the assistant generate answers, have the reward model score them, and use reinforcement learning to nudge the assistant toward higher-scoring (more-preferred) answers — while a KL leash keeps it from drifting too far from the sensible SFT model.

First we gather the raw material — preference data. The recipe is simple and repeats thousands of times:

1. Take a prompt.
2. Have the model generate two (or more) candidate responses.
3. A human labeler reads them and picks the better one.

Each judgment becomes a preference pair: a chosen (preferred) response and a rejected one for the same prompt.

Prompt:   "Explain why the sky is blue, simply."
Response A: "Sunlight is white, but air scatters its blue
             light more than other colors, so the sky
             looks blue."                        ← human picks this  ✅ chosen
Response B: "Because of the atmosphere and physics
             and stuff."                          ← ❌ rejected

Notice how little effort that judgment took versus writing a perfect explanation. That asymmetry is what lets RLHF scale.

We can't ask a human to grade every answer during training — that would take millions of judgments. So we train a stand-in: the reward model (RM).

The RM is itself a language model (often a copy of our model with a small scoring head bolted on) that takes a (prompt, response) and outputs a single number — a reward score. We train it on the preference pairs with one objective: give the chosen response a higher score than the rejected one.

After training on enough pairs, the RM has effectively learned human taste: hand it any new response and it predicts how much a human would like it — instantly, and a million times over. It becomes the automatic judge that makes the next step possible.

Mental model: humans teach their preferences to the reward model once; the reward model then grades the assistant tirelessly during RL.

Now the actual learning. We treat the assistant as a policy and run a reinforcement-learning loop (the classic algorithm is PPO, Proximal Policy Optimization):

1. The assistant generates a response to a prompt.
2. The reward model scores it.
3. PPO updates the assistant's weights to make high-scoring responses more likely and low-scoring ones less likely.
4. Repeat, millions of times.

There's one essential safeguard. If you let the model only chase reward, it cheats — it discovers weird, degenerate text that the RM happens to score high but humans would hate (more on this next). So PPO adds a KL-divergence penalty: a leash that punishes the policy for drifting too far from the original SFT model.

The picture to hold: "climb toward higher reward, but don't wander far from home." Reward pulls the model toward what humans like; the KL leash keeps it fluent, sane, and recognizably itself.

The reward model is only a proxy for real human preference — and optimizers are ruthless at exploiting proxies. This is reward hacking (a.k.a. reward over-optimization): the policy finds responses that score high on the RM without actually being better.

Classic symptoms:

• Verbosity — longer answers often correlate with "better" in the training data, so the model learns to pad.
• Sycophancy — agreeing with the user / telling them what they want to hear scores well, even when it's wrong.
• Format tricks — confident tone, bullet points, or flattery that look high-quality to the RM.

This is a genuinely open problem and a big reason model "personalities" can feel over-eager or over-formatted. Defenses: the KL leash, better and bigger reward models, fresh preference data, and capping how hard you optimize. Knowing reward hacking exists explains a lot of real model quirks — exactly the kind of nuance most courses skip.

Human preference labels are powerful but slow, expensive, and inconsistent (~$1+ per judgment, and people disagree). The modern fix: let an AI do much of the labeling.

• RLAIF (RL from AI Feedback): instead of a human picking A vs. B, a capable model judges the pair, guided by written guidelines. By 2025 this matched RLHF quality at a small fraction of the human-labeling cost (commonly cited as a ~100× reduction) — so it's now a default for large-scale preference data.
• Constitutional AI (Anthropic's approach): give the AI judge an explicit written "constitution" — principles like be helpful, avoid harm, be honest — and have the model critique and revise its own answers against those rules. This bakes harmlessness in transparently, from stated values rather than millions of opaque human clicks.

The catch (be honest): an AI judge inherits its own biases and blind spots, so humans still anchor and audit the process. But the direction is clear — scalable preference signals, with humans steering rather than hand-labeling everything.

You'll almost never run PPO yourself — but RLHF shapes the models you use every day, and understanding it pays off:

• It's why models feel helpful, safe, and well-toned — that polish is RLHF, not pretraining.
• It explains real failure modes you'll hit: sycophancy ("you're absolutely right!" when you're not), over-refusal, and verbosity — all fingerprints of preference optimization and reward hacking.
• Provider "vibes" differ because each lab makes different preference and constitution choices — same idea behind why Claude, GPT, and Gemini feel distinct.
• You may use preference data yourself — collecting (chosen, rejected) pairs for your task is increasingly practical, especially with the simpler method coming next.
• It sets up DPO — most teams today skip the reward-model-plus-PPO machinery for a lighter approach that learns from the same preference pairs directly.

Be the labeler. A model answers a question two ways: A) a clear, correct, friendly paragraph; B) a curt, partly-wrong reply. You pick A.

That single A-over-B choice is exactly the data that trains the reward model in RLHF. Question: why is ranking A vs B easier (and more scalable) than writing the perfect answer yourself? → Because judging 'which is better' takes seconds, while authoring the ideal answer is slow and hard — that asymmetry is what makes RLHF work.

• "RLHF teaches the model new facts." No — like all post-training, it shapes behavior/preference, not knowledge. It steers ability the model already has.
• "The reward model is a human." No — it's a learned proxy trained once on human comparisons, then used to grade millions of responses automatically (and it can be gamed).
• "More reward is always better." No — past a point you get reward hacking; the KL leash deliberately limits how hard you optimize.
• "RLHF needs humans for everything." Increasingly no — RLAIF / Constitutional AI let an AI judge handle most labeling, with humans steering.
• "RLHF = PPO." PPO is the classic optimizer, but the goal is learning from preferences — and DPO (next) reaches it without RL at all.

• RLHF fixes SFT's ceiling by learning from comparisons ("which answer is better?") instead of single demonstrations — richer feedback, easier to collect.
• Three steps: start from the SFT model → train a reward model on human preference pairs → optimize with RL (PPO) toward higher reward, held by a KL leash to the SFT model.
• The reward model is a learned, automatic stand-in for human taste — which is also why it can be reward-hacked (verbosity, sycophancy, format tricks).
• RLAIF / Constitutional AI scale preferences cheaply by using an AI judge guided by written principles, with humans steering.
• RLHF is why assistants feel helpful and safe — and why they sometimes flatter, refuse, or ramble.

Next: DPO — Direct Preference Optimization — which reaches the same goal from the same preference pairs, but drops the separate reward model and the RL loop entirely, making preference tuning simple enough to do yourself.