Reasoning Fine-Tuning with GRPO

Introduction

L202 (DPO) taught a model what people prefer. This lesson teaches it to reason — and it's the method behind the reasoning-model explosion of 2026: GRPO (Group Relative Policy Optimization), the algorithm that powers DeepSeek-R1.

Here we finally cross from preference tuning into real reinforcement learning — but a beautifully simplified kind. Classic RLHF needed a reward model and a critic; GRPO throws both away. The trick: instead of learning a reward, you verify correctness directly (math checker, unit test), and instead of a critic estimating a baseline, you let a group of the model's own attempts be the baseline. The result is RLVR — Reinforcement Learning with Verifiable Rewards — and it's how a model grows the ability to think step by step, including the famous emergent "aha moment." By the end you'll know the GRPO loop, how to design the reward, and the three ways it fails.

An infographic titled 'Reasoning Fine-Tuning with GRPO', the third lesson of section four on advanced post-training, covering the reinforcement-learning method that teaches models to reason. GRPO, Group Relative Policy Optimization, is the algorithm behind DeepSeek-R1. Its loop is: for each prompt, such as a math problem, the model samples a whole GROUP of completions, often eight or sixteen attempts; each completion is scored by a verifiable reward function, for example a math checker or a unit test that simply checks whether the answer is correct; then the group-relative advantage of each completion is computed by normalizing its reward against the group, advantage equals reward minus the group mean divided by the group standard deviation; and finally a PPO-like clipped update with a KL penalty to a reference model nudges the policy toward the high-advantage completions and away from the low ones. The defining trick is that the group's own average is the baseline, so GRPO needs no separate critic or value model, which roughly halves the memory of PPO. Because the reward comes from an automatic verifier rather than a learned reward model or human labels, this is reinforcement learning with verifiable rewards, RLVR, and it works on any task whose correctness can be checked. Reward design is the craft: DeepSeek used a rule-based accuracy reward for correctness plus a format reward for putting reasoning in think tags and the answer in answer tags. The remarkable result is emergence: DeepSeek-R1-Zero, trained with pure RL and no supervised fine-tuning, spontaneously grew long chain-of-thought, self-correction, and the famous aha moment. The failure modes are reward hacking, where the reward rises while real performance falls, so side evaluations independent of the training reward are essential; length bias, which Dr. GRPO fixes by removing the length normalization; and the subtle no-signal case where every rollout in a group earns the same reward, making every advantage zero, since GRPO learns only from the spread within a group. The takeaway is that GRPO grows reasoning by sampling many attempts and rewarding what verifiably works, with the group as its own baseline.

The GRPO Loop: Sample a Group, Reward What Works

GRPO is online RL — the model learns from data it generates itself. The loop, for one prompt:

Sample a group. For a prompt (say a math problem), the model generates a group of G completions — typically 8–16 different attempts (rollouts), each a full reasoning chain + answer.
Score each with a verifiable reward. A rule-based reward function checks each completion — is the final answer correct? (a math verifier), do the unit tests pass? (a code runner). No human, no learned reward model.
Compute the group-relative advantage. Normalize each reward against the group's own statistics:

Âᵢ = (rᵢ − mean(r)) / std(r)

A completion that beats the group average gets a positive advantage; one below average gets negative. The group mean is the baseline.
Update the policy. A PPO-style clipped objective nudges the model to make high-advantage completions more likely and low-advantage ones less likely — with a KL penalty to a reference model for stability.

Repeat over thousands of prompts and the model does more of what verifiably works — it learns to reason. You'll run this exact loop, by hand, in the lab.

The Key Trick: The Group Is Its Own Baseline

Why is GRPO such a big deal versus PPO (classic RLHF's RL algorithm)? It deletes the critic.

PPO needs a separate value (critic) model running alongside the policy to estimate a baseline — how good is this state on average? — so it can tell whether a given completion was better or worse than expected. That critic is another full-size model in memory, roughly doubling the training cost.

GRPO's insight: you don't need a learned baseline if you sample a group. The average reward of the group is the baseline. A completion's advantage is just how far above or below its siblings it scored. So GRPO is critic-free — same idea as PPO (push toward better-than-baseline outcomes), but the baseline comes from the group, not a value network. Half the memory, simpler, and it matches or beats PPO on reasoning.

One profound consequence you'll feel in the lab: if every rollout in a group gets the same reward (all correct, or all wrong), the std is ~0, so every advantage is 0 — and the model learns nothing from that group. GRPO lives on the spread within a group.

RLVR: Rewards You Can Verify, Not Label

The other half of GRPO's power is where the reward comes from. Classic RLHF/DPO needs preference data — humans (or an AI) comparing outputs. GRPO for reasoning uses RLVR — Reinforcement Learning with Verifiable Rewards — where the reward is an automatic, programmable check:

Math: is the final answer equal to the known solution? (a symbolic/numeric checker).
Code: do the unit tests pass? (run them).
Logic/format: does the output match the required structure?

This is transformative: no labels, no reward model to train (or game) — just a verifier. It works on any task whose correctness you can check programmatically. That's exactly why GRPO is the biggest lever for reasoning (math, code, logic) and not for subjective qualities like tone (you can't verify "on-brand" — that's DPO's job, L202). The decision rule from L201: verifiable reward → GRPO; preference pairs → DPO.

Designing the Reward (and the Emergent "Aha Moment")

The reward function is the craft of GRPO — it's the only thing telling the model what "good" means. DeepSeek-R1 used two simple rule-based rewards, summed:

Accuracy reward — +1 if the final answer is correct. This is the real signal.
Format reward — a small bonus for putting reasoning in <think>…</think> and the answer in <answer>…</answer>. This shapes the output into structured thinking.

In TRL a reward function is just Python: it takes the completions and returns a list of scalars (you can pass several and they're summed). Keep accuracy dominant — more on why in the failure modes.

The astonishing payoff: DeepSeek-R1-Zero was trained with pure GRPO and no SFT first — and it spontaneously developed long chain-of-thought, self-correction ("wait, let me reconsider…"), and the famous "aha moment." The hypothesis: forcing human reasoning patterns via SFT can limit exploration, while unrestricted RL on a verifiable reward lets novel reasoning emerge. Reasoning wasn't taught — it was incentivized into existence.

The Full Pipeline & The Failure Modes

R1-Zero proved pure RL works, but its output was messy (mixed languages, unreadable). The production DeepSeek-R1 wraps GRPO in a pipeline: a little cold-start SFT (for readable output) → reasoning RL with GRPO → rejection sampling + SFT (keep the best generations, retrain) → a final RL for helpfulness/safety. GRPO is the reasoning engine inside the L201 stack.

Three failure modes you must watch for:

Reward hacking — the model finds a way to inflate the reward without actually reasoning. DeepSeek saw training reward rise while real CodeForces performance fell. If you over-weight the format reward, the cheapest win is to emit the tags, not to think. Fix: keep accuracy dominant, and always run side evals independent of the training reward.
Length bias — original GRPO normalizes the loss by sequence length, so short correct answers get disproportionately large gradients (or long ones get diluted) — biasing the model toward terseness/verbosity regardless of quality. Dr. GRPO removes the length normalization.
No-signal groups — if a group is all-correct or all-wrong, there's no variance → zero advantage → no learning. Pick problems at the edge of the model's ability so attempts actually differ.

(Other variants: DAPO — decoupled clipping, dynamic sampling, drops the KL term — for more exploration and stability at scale.)

In Code: GRPO with TRL

In TRL, GRPO is just a reward function + the trainer. You write the verifier in plain Python; TRL handles the group sampling, the advantage normalization, and the clipped update:

from trl import GRPOTrainer, GRPOConfig

# A VERIFIABLE reward = a plain function: (prompts, completions) -> list[float].
def accuracy_reward(completions, answer, **kwargs):     # the REAL signal
    return [1.0 if extract_final(c) == a else 0.0       # a checker, not a model
            for c, a in zip(completions, answer)]

def format_reward(completions, **kwargs):               # a SMALL shaping bonus
    return [0.2 if ('<think>' in c and '<answer>' in c) else 0.0 for c in completions]

trainer = GRPOTrainer(
    model=model,                                        # often after a cold-start SFT
    reward_funcs=[accuracy_reward, format_reward],      # summed; keep ACCURACY dominant
    train_dataset=math_prompts,
    args=GRPOConfig(
        num_generations=8,        # GROUP SIZE G - the rollouts per prompt (the baseline)
        beta=0.04,                # KL penalty to the reference (stability)
        # no value/critic model exists - the group average IS the baseline
    ),
)
trainer.train()   # ALSO track a SIDE eval - reward up != skill up (reward hacking).

See It: The GRPO Lab

Run a GRPO step by hand. The model has sampled a group of attempts at 17 × 23. Now design the reward: weight accuracy vs format, set the group size G, and watch each rollout get a reward and a group-relative advantage (↑ above the mean baseline, ↓ below). Then break it on purpose: crank format ≥ accuracy to see reward hacking; turn on length-normalize to see length bias; set accuracy = 0 and a group that's all-correct to see the no-signal collapse (every advantage → 0).

Interactive: the GRPO Lab. For the prompt seventeen times twenty-three, the model samples a group of rollouts, each shown with its reasoning trace and final answer. The user designs the verifiable reward by weighting an accuracy reward and a format reward, toggling length-normalization between original GRPO and Dr. GRPO, and setting the group size and the KL leash. The lab then scores every rollout, computes the group mean as the baseline, and shows each rollout pushed up with a positive advantage or down with a negative one, where the advantage equals reward minus mean divided by standard deviation. A verdict surfaces the three classic failure modes live: reward hacking when the format reward meets or beats accuracy so the cheapest win is to emit the tags rather than reason; length bias when length normalization is on and short correct rollouts get oversized updates; and the no-signal case where the whole group earns the same reward so every advantage collapses to zero and there is nothing to learn, illustrating that GRPO depends on variance within the group. A pipeline strip with arrows shows prompt to sample G rollouts to verifiable reward to advantage.

The whole algorithm is right there: reward what verifiably works, measured against the group's own average — and the failures all come from a badly-shaped reward or a group with no spread.

Why This Matters

GRPO is why 2026 has reasoning models at all. It's the method that turns a competent model into one that can work through multi-step math, debug code against tests, and check its own work — capabilities no amount of SFT or DPO can instill, because they require learning from outcomes, not imitation or preference. For an AI engineer, GRPO is the lever you reach for when the task has a verifiable answer and the model needs to get it right, not just sound good. And understanding its mechanics — group baseline, verifiable reward, the reward-hacking and no-signal traps — is what lets you actually run it without burning a GPU cluster on a model that learned to game your metric. Next, L204 (Model Merging — SLERP, TIES, DARE) shows how to combine the specialist models this section produces.

🧪 Try It Yourself

Drive GRPO and trigger every failure in the GRPO Lab:

A healthy step. Accuracy ×1.0, format ×0.2, G=8. Which rollouts get positive advantage and which negative — and what is the baseline they're compared against (no critic model in sight)?
Reward hacking. Push format ≥ accuracy. Why will the model learn to emit tags instead of reasoning, and what real-world signal must you watch besides the training reward?
No signal. Make every rollout score the same (e.g. accuracy=0, format=0, or a group that's all correct). Why does advantage = 0 for everyone, and what does that teach you about choosing problems?

Bonus: GRPO uses no critic and no reward model. In one sentence each: what plays the role of the critic's baseline, and what plays the role of the reward model?

Mental-Model Corrections

"GRPO needs a reward model like RLHF." No — it uses a verifiable reward (a checker/verifier), not a learned reward model. That's RLVR.
"GRPO needs a critic/value model like PPO." No — the group's average reward is the baseline. Critic-free is the whole point (half the memory).
"More reward = better model." Beware reward hacking — the reward can rise while real skill falls. Always run side evals.
"Any group of rollouts teaches something." No — an all-correct or all-wrong group has zero variance → zero advantage → no learning. GRPO needs a spread.
"GRPO can align tone/style." Not really — it needs a verifiable reward (math, code). Subjective alignment is DPO's job (L202). Use the right tool for the signal you have.

Key Takeaways

GRPO loop: sample a group of G rollouts → score each with a verifiable reward → advantage Â = (r − mean)/std → clipped update with a KL leash. It teaches reasoning.
The group is its own baseline → no critic model (≈ half PPO's memory). And if a group has no variance, there's no learning signal.
RLVR: the reward is an automatic verifier (math checker, unit tests) — no human labels, no reward model. Works on any checkable task.
Reward design = accuracy (+ a small format bonus); pure RL can make reasoning emerge (R1-Zero's "aha moment").
Failure modes: reward hacking (run side evals), length bias (Dr. GRPO), no-signal groups (pick edge-of-ability problems).

Next: L204 — Model Merging (SLERP, TIES, DARE) — combining the specialist models you've now learned to train.