Reasoning Models & Test-Time Compute

Introduction

Welcome to The Frontier — the final section of the course. Everything until now taught models to perceive and act: read text, see images, hear speech, retrieve knowledge, call tools. This section turns to the cutting edge of how models think and operate: deliberate reasoning, using a computer and browser, writing code as agents, the open research problems, and how to keep up. We start with the idea that reshaped the field in 2025–2026: reasoning models and test-time compute.

Here's the shift in one sentence. For years, the way to get a smarter model was to make it bigger — more parameters, more training data (the pre-training scaling laws). Around late 2024, a second lever appeared: let the model spend more compute at inference — think longer before it answers — and accuracy on hard problems climbs dramatically. A model that 'blurts' an answer in one pass is replaced by one that explores, checks its work, backtracks from dead ends, and verifies — visibly thinking first. OpenAI's o-series, DeepSeek-R1, Google's Gemini thinking, and Claude's extended thinking are all this idea.

For you as an engineer, reasoning is a new dial on the control panel. Used well, it turns a 22%-on-a-hard-proof model into a 90% one. Used carelessly, it pays 10× the cost and 10× the latency to write a tagline no better than before. This lesson is about the mechanism and the judgement.

In this lesson:

CoT prompting vs. reasoning models — a prompt trick vs. a trained capability.
The second scaling law — test-time (inference) compute, and why it works.
How reasoning models are trained — RL on verifiable rewards (the DeepSeek-R1 story).
Sequential vs. parallel test-time compute — think longer or think wider.
Using extended thinking — the Claude API (budgets, effort, billing, interleaved thinking).
When reasoning helps — and when it hurts (overthinking).
Reasoning + tools = agents, and evaluating reasoning.

Scope: this builds on the scaling-law and evaluation (L246) lessons, on chain-of-thought prompting from the prompting section, and on the agents material. It defers the rest of The Frontier: computer & browser use → L257, coding agents → L258, open research problems → L259, staying current → L260. Here we focus on reasoning itself and the compute that powers it.

Hero infographic titled 'Reasoning Models & Test-Time Compute' for lesson L256, the opening lesson of the section 'The Frontier: Reasoning, Computer/Browser Use & Beyond', on a white background. The deck says the course turns from how models perceive to how they THINK: reasoning models spend extra compute at inference to deliberate before answering. The left panel, 'A SECOND SCALING LAW', shows two axes meeting: the old axis is pre-training compute (more data and parameters), the new axis is test-time compute (spend more thinking at inference); a small curve shows accuracy rising logarithmically with thinking tokens, with a striking note that a small model with about 100 times more inference compute can match a model ten times its size. Below it a contrast: chain-of-thought PROMPTING (a 'think step by step' prompt on a standard model) versus a REASONING MODEL (reinforcement-learning trained to deliberate natively, with verifiable rewards on math and code, as in DeepSeek R1's pure-RL training where reflection and backtracking emerged on their own). The right panel, 'HOW TO SPEND THE COMPUTE', shows two strategies side by side: SEQUENTIAL — one long chain (explore, check, backtrack, verify) that buys accuracy with latency; and PARALLEL — sample N chains and pick the winner by majority vote (self-consistency) or a verifier (best-of-N), which buys accuracy with cost but little extra latency. A control strip shows the Claude API knobs: extended thinking with a token budget, or adaptive thinking with an effort level, and the warning that thinking tokens bill as output tokens and add latency. A benchmarks note lists AIME, GPQA Diamond, ARC-AGI-2 and FrontierMath as the tests that separate reasoning models. Three cards along the bottom read: 'Compute is the new scale' (spend at inference, logarithmic gains, diminishing returns); 'Longer vs wider' (sequential trades latency, parallel trades cost); and 'Use it where it pays' (huge on hard multi-step problems, wasteful overthinking on easy ones — match the spend to the cost of a wrong answer). A family strip lists the new section: reasoning & test-time compute L256, computer & browser use L257, coding agents L258, open research problems L259, staying current L260.

Chain-of-Thought Prompting vs. Reasoning Models

These two are constantly confused, and the difference is the heart of the lesson.

Chain-of-thought (CoT) prompting is a technique: you ask a standard model to "think step by step," and it produces intermediate reasoning before its answer, which improves accuracy on multi-step problems. It's a prompt you write. The model wasn't specially trained for it; you're coaxing latent ability out with words. (You met this in the prompting section.)

Reasoning models bake that behavior into the weights. They are post-trained (with reinforcement learning) to deliberate natively — to generate a long internal chain-of-thought that explores, self-checks, and backtracks, before emitting the visible answer. You don't have to prompt for it; it's a trained capability, and it goes far deeper than a 'think step by step' nudge — these models will happily spend thousands of tokens working a problem out on internal scratch paper.

Reasoning models expose that deliberation in different ways, and it's worth knowing the four flavors:

Hidden — the reasoning happens in tokens you never see (OpenAI o3/o4-mini): you pay for them but only get the answer.
Inspectable & budgeted — a separate, controllable thinking block (Claude extended thinking): you set a budget and can read a summary.
Streamed inline — a transparent chain shown as it's produced (DeepSeek-R1).
Parallel — multiple hypotheses explored at once (Gemini Deep Think-style).

A useful rule of thumb: CoT prompting is free and works on any model but is shallow; a reasoning model is a trained, deeper, costlier capability you switch on for hard problems. The rest of this lesson is about the compute that makes the trained version so powerful.

The Second Scaling Law — Test-Time Compute

The pre-training scaling laws (the Chinchilla-era result) said: to lower loss, scale parameters and data together — bigger models trained on more tokens are predictably better. That's training-time compute, paid once, up front.

Test-time (inference-time) compute is a second, independent scaling axis: hold the model fixed and spend more compute per query — longer reasoning chains, multiple attempts, self-verification — and accuracy keeps rising. The headline result that stunned the field: on hard reasoning tasks, accuracy improves roughly logarithmically with the number of 'thinking' tokens. And the trade against model size is real — a small model given ~100× more inference compute can match a model ~10× its size run normally. You can, in effect, buy intelligence at inference instead of at training.

Why does thinking longer help at all? Because a single forward pass is a fixed amount of computation, but many real problems need variable amounts of work — a hard integral or a tricky proof needs more 'steps' than a greeting. Letting the model roll out a long chain (and revisit it) lets it spend computation proportional to difficulty, the way a person reaches for scratch paper on a hard problem but answers an easy one instantly.

Two consequences you'll feel as an engineer:

It's a dial, with diminishing returns. Because gains are logarithmic, the first few thousand thinking tokens help a lot and the next ten thousand help a little. There's a knee past which you're paying for almost nothing — you'll find it in the lab.
It moves cost from training to inference. Industry forecasts put inference compute demand far above training by 2026 — the economics of running these models are now dominated by how much you let them think.

How Reasoning Models Are Trained — RL on Verifiable Rewards

Reasoning models aren't just bigger; they're trained to reason — and the recipe is reinforcement learning on problems where the answer can be automatically checked.

The insight: for math (is the final number correct?) and code (do the tests pass?), you don't need a human to grade — you have a verifiable reward. So you let the model attempt a problem, generate a long chain-of-thought, and reward it when the final answer is right. Over many iterations, the model learns the reasoning behaviors that lead to correct answers.

The DeepSeek-R1 result made this vivid. DeepSeek-R1-Zero was trained with pure RL on the base model — no supervised chain-of-thought examples at all — using GRPO (Group Relative Policy Optimization) with simple rule-based accuracy rewards. The striking finding: reasoning behaviors like reflection, double-checking, and exploring alternative approaches emerged on their own — the model taught itself to deliberate (the so-called 'aha moment'), purely because deliberation earned reward. OpenAI's o-series and Anthropic's extended thinking use related RL-based approaches (plus some supervised CoT data for polish).

Two ideas from this carry into how you use reasoning models (next sections):

Outcome vs. process rewards. You can reward only the final answer (outcome) or each step of the reasoning (a process reward model / verifier). Verifiers also power inference-time tricks like best-of-N (pick the candidate the verifier scores highest).
Verifiable domains lead. Reasoning models are strongest where correctness is checkable (math, code, logic) and weaker where it isn't (taste, open-ended writing) — which is exactly the when-to-use line.

Sequential vs. Parallel — Think Longer or Think Wider

Once you've decided to spend compute at inference, there are two ways to spend it — and they trade off differently. The lab lets you feel both.

Sequential — think longer. Generate one long chain of thought: attempt, check, notice a mistake, backtrack, try another route, verify. More thinking tokens → a deeper single line of reasoning. This is what 'extended thinking' does. It buys accuracy at the cost of latency — the chain is serial, so a big budget means seconds (sometimes tens of seconds) before the first answer token.

Parallel — think wider. Sample N independent attempts at once and pick the best. Two ways to pick:

Self-consistency (majority vote): take the answer that the most chains agree on. No extra model needed — surprisingly strong on math.
Best-of-N (verifier): score each candidate with a reward model / verifier and keep the highest. Better than majority when the verifier is reliable.

Parallel buys accuracy at the cost of compute/$ (you run the model N times) — but, crucially, with little extra latency, because the N runs go concurrently. So the choice is a real engineering trade: sequential = pay in time; parallel = pay in money.

And you can combine them — sample several long chains and verify across them — which beats either alone on the hardest problems. The mental model: sequential deepens one path; parallel explores many and selects. In the lab, switch between them on the same hard problem and watch the latency tile (grows for sequential, flat for parallel) and the cost tile (linear in budget vs. linear in N).

Using Extended Thinking — the Claude API

Turning reasoning on is a single parameter. With Claude extended thinking, you give the model a budget of internal reasoning tokens; it thinks within that budget, then answers:

import anthropic
client = anthropic.Anthropic()

resp = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=16000,
    thinking={"type": "enabled", "budget_tokens": 4000},   # >= 1024 and < max_tokens
    messages=[{"role": "user",
               "content": "Are there infinitely many primes p with p % 4 == 3? Prove it."}],
)

for block in resp.content:
    if block.type == "thinking":
        print("THINKING:", block.thinking)   # the deliberation (+ a signature to pass back)
    elif block.type == "text":
        print("ANSWER:", block.text)

# Billing: the FULL thinking tokens are charged at the OUTPUT rate. ~4k thinking before a
# ~500-token answer costs roughly 9x the bare answer — so reserve it for problems that need it.

Three things to know:

Adaptive thinking & effort (current flagships). On the newest models (Claude Opus 4.8, Sonnet 4.6), thinking is adaptive — instead of a fixed budget_tokens, the model decides how much to think per request, and you steer it with an effort level (e.g. low → medium → high → max; default high). Simple prompts get fast answers; hard prompts get deep reasoning — automatically. (Explicit budget_tokens is the Claude 4.5-era control.)
Billing & latency. Thinking tokens bill as output tokens (so a big budget multiplies cost), and time-to-first-visible-token goes from milliseconds to ~3–15 seconds with the budget. The response returns thinking blocks (a summary plus a cryptographic signature) before the text answer.
Interleaved thinking with tools. Reasoning models can think between tool calls, not just before the first response — reasoning about each tool result before the next action. When you continue such a conversation you must pass the thinking blocks back unchanged with the tool result, or the API errors. This is what makes reasoning models powerful agents — the next idea.

When Reasoning Helps — and When It Hurts

Reasoning is not a free upgrade you flip on everywhere. It is a component with a price, and the single most useful judgement you'll make is where it pays.

It helps — a lot — when the problem needs multi-step work:

Competition-style math and physics (multiple unknowns, long derivations).
Code that must be correct (debugging, algorithmic problems, tricky edge cases).
Logic, planning, and optimization — anything where a person would reach for scratch paper.
Careful analysis where a subtle mistake is costly. The rule: if a wrong answer is expensive to fix, buy the thinking.

It hurts — or just wastes money — on simple tasks:

Formatting, summarising, writing copy, casual chat, simple extraction. A standard model already nails these; thinking adds latency and cost for ~zero accuracy gain. This is overthinking — and on some easy tasks, forcing long deliberation can even degrade the answer.

Two principles for production:

Match the spend to the cost of a wrong answer. High-stakes, hard, verifiable → reason (and maybe best-of-N). Routine, cheap-to-redo → a fast standard model.
Reasoning models are components, not a default. Swapping a reasoning model in for everything pays the cost without capturing the benefit. Many teams route: a fast model handles the bulk, and a reasoning model is called only for the hard sub-problems (or as the planner) — which is exactly the agentic pattern next.

Reasoning + Tools = Better Agents

Reasoning matters most when the model can act on the world between thoughts — which is why it supercharges agents (and bridges directly to the rest of this section).

Recall the agent loop: think → call a tool → observe the result → think again. A reasoning model with interleaved thinking reasons at every step — not just once at the start, but about each tool result, deciding what the new information means and what to do next. For multi-step tasks where each result changes the plan, that 'reason between actions' shape is a different, more reliable capability than single-shot reasoning. (Mechanically, this is the interleaved-thinking behavior from the API section — and why you must preserve the thinking blocks across turns.)

In practice the dominant pattern is a division of labor:

A reasoning model as the planner/reflector — it decides the strategy, reflects on failures, and handles the genuinely hard sub-steps.
Faster, cheaper models (or plain tool calls) do the bulk of the per-step work.

Some 2026 models even switch modes within a single model — fast 'non-thinking' responses for easy turns, deep thinking for hard ones — to get the benefit without paying for it everywhere. This is the foundation for computer & browser use (L257) and coding agents like Claude Code (L258): both are reasoning models driving tools in a loop, reasoning about what they see after each action.

Evaluating Reasoning — Benchmarks & Faithfulness

How do you know a reasoning model is actually reasoning and not pattern-matching? With benchmarks designed to resist memorization — and a healthy skepticism about the reasoning trace itself.

The benchmarks that separate 2026 models:

AIME (competition math) — frontier models now near-saturate it (a top 2026 model solved all 30 AIME-2025 problems).
GPQA Diamond (PhD-level science questions, 'Google-proof') — the low-90s% for the best models.
ARC-AGI / ARC-AGI-2 (abstract visual puzzles built to defeat memorization) — still hard: even frontier models score far below humans, which is why it's a favorite frontier test.
FrontierMath (research-level math, designed to resist memorization) — the hardest public math set; where models genuinely separate.

The pattern: as models memorize old benchmarks, the field builds harder, novel ones (FrontierMath, Humanity's Last Exam) that force actual multi-step reasoning over recall.

The faithfulness caveat (important and honest). The visible chain-of-thought is not guaranteed to be a faithful account of how the model actually reached its answer — research shows models sometimes reach a conclusion and then produce plausible-looking reasoning, or omit the real influence on their answer. So: treat the thinking trace as a useful artifact (great for debugging and steering), not as a verified audit log of the model's true computation. And keep cost in your evals — a model that scores 2 points higher for 5× the thinking budget may not be worth it. Tie it back to L246: evaluate reasoning on your tasks, scoring accuracy and the compute it took.

See It — The Test-Time Compute Lab

Time to spend compute and watch the trade. The lab starts on a hard AIME-style proof with sequential thinking at a modest budget. Drag the thinking budget up and watch the chain grow (read → plan → attempt → check → backtrack → verify) and accuracy climb — then notice the knee where more budget barely helps (diminishing returns) and the latency tile keeps rising.

Now switch to Parallel (best-of-N) on the same problem: sample N chains and pick by majority vote or a verifier. Watch accuracy rise with N while latency stays flat but cost scales with N — the opposite trade from sequential. Toggle majority vs. verifier to see the verifier edge.

Finally, the punchline: switch the task to the trivial tagline and keep thinking on. Accuracy barely moves (it was already ~97%) but you're paying many times the cost — the verdict flips to overthinking. That single comparison is the whole judgement of this lesson.

The Test-Time Compute Lab — spend compute at inference and watch the trade. Pick a TASK (a hard AIME-style proof, a multi-step word problem, or a trivial tagline) and a STRATEGY: none (a standard model answers in one pass), SEQUENTIAL (extended thinking — drag the thinking-token budget and watch one long chain grow and accuracy climb logarithmically while latency and cost rise), or PARALLEL (best-of-N — sample N chains and pick the winner by majority vote or a verifier). A live reasoning-trace diagram shows the difference; accuracy / compute-cost / latency tiles and a verdict update. The lessons land: reasoning lifts the hard proof from ~22% to ~90% but adds ~1 point of 'overthinking' to the trivial tagline for many times the cost; returns diminish past the knee; sequential buys accuracy with LATENCY, parallel buys it with COST. Match the spend to the cost of a wrong answer. Illustrative.

Notice three things. One: on the hard problem, thinking is transformative (~22% → ~90%); on the easy one it's pure waste. Two: accuracy rises logarithmically — the knee is real, so unbounded budgets are a trap. Three: sequential pays in latency, parallel pays in cost — pick the one your product can afford.

🧪 Try It Yourself

Predict first, then check in the lab (or reason it out).

1. You wrap your customer-support FAQ bot in a reasoning model with a 16k thinking budget "to make it smarter." Latency triples, costs 8×, and answer quality is unchanged. What happened, and what's the fix?

2. A model gets a hard math problem right 30% of the time in one shot. You sample 8 answers and keep the most common one, and accuracy jumps to ~75% with no verifier. What technique is this, and why does it work?

3. Two setups reach the same accuracy on a hard task: (a) one long 20k-token chain, (b) 10 parallel 5k-token chains + a verifier. Your product is a live chat with a strict latency SLA. Which do you ship, and why?

4. Your reasoning model's visible chain-of-thought says it used formula X — but you suspect it actually pattern-matched a memorized answer. Can you trust the trace as proof? What does that imply for using CoT as an audit log?

5. When does it make sense to put a reasoning model as the planner of an agent while a cheaper model does the steps, rather than making the whole agent a reasoning model?

Answers.

1. Overthinking — FAQ answers are easy, so deliberation adds latency and cost for ~zero accuracy gain. Fix: use a fast standard model for the bot; reserve reasoning (or adaptive thinking with a low effort) for the rare genuinely-hard ticket. Reasoning is a component, not a default.

2. Self-consistency (majority vote) — a form of parallel test-time compute. Sampling many chains and taking the majority answer works because correct reasoning paths tend to converge on the same answer while wrong ones scatter — so the mode is far more reliable than a single sample, without needing a verifier.

3. (b), the parallel + verifier setup. The N chains run concurrently, so latency ≈ one 5k chain — far under a 20k serial chain that would blow the SLA. You pay for it in compute/$, not time. (If cost were the constraint and latency loose, you'd flip the choice.)

4. No — the chain-of-thought is not a guaranteed-faithful audit log. Models can reach an answer and then generate plausible-sounding reasoning, or hide the real influence. Treat the trace as a useful debugging artifact, not proof; if you need verifiable correctness, check the output (run the code, verify the math), don't just trust the explanation.

5. When most steps are easy and only the strategy is hard — let an expensive reasoning model plan and reflect (and handle the hard sub-problems) while a cheap model executes routine steps. You get the reasoning benefit on the part that needs it and pay fast-model prices for the bulk — far cheaper than reasoning over every single turn.

Mental-Model Corrections

"A reasoning model is just a model with a 'think step by step' prompt." → CoT prompting is a trick on a standard model; a reasoning model is RL-trained to deliberate natively, far deeper, with thousands of internal tokens.
"You only get smarter models by making them bigger." → Test-time compute is a second scaling axis: spend more at inference and accuracy climbs (logarithmically). A small model with lots of inference compute can match a much larger one.
"More thinking is always better." → Gains are logarithmic with diminishing returns past a knee — and on easy tasks thinking is overthinking (cost & latency for nothing).
"Sequential and parallel are the same thing." → Sequential = one long chain (pay in latency); parallel = N chains + a selector (pay in cost, ~flat latency). Different trades.
"Best-of-N needs a verifier." → Self-consistency (majority vote) needs no verifier and is strong on math; a verifier (best-of-N) does better when it's reliable.
"The visible chain-of-thought is what the model actually did." → CoT is not guaranteed faithful — it can be post-hoc. Useful for steering/debugging; not a verified audit log.
"Reasoning tokens are cheaper than output." → They bill as output tokens and add seconds of latency. Budget accordingly.
"Make the whole agent a reasoning model." → Usually wasteful. Route: a reasoning model plans/reflects and handles hard sub-steps; cheap models do the bulk.

Key Takeaways

Reasoning models are RL-trained to deliberate before answering (verifiable rewards on math & code; DeepSeek-R1 showed reasoning behaviors emerge from pure RL). That's different from CoT prompting, a shallow trick on a standard model.
Test-time compute is a second scaling law: hold the model fixed, spend more compute at inference, and accuracy rises logarithmically — a small model with ~100× inference compute can match one ~10× its size. Inference now dominates the compute bill.
Two ways to spend it: sequential (one long chain — explore/check/backtrack/verify; pay in latency) and parallel (sample N, pick by majority vote / self-consistency or a verifier / best-of-N; pay in cost, ~flat latency). Combine them for the hardest problems.
The API: Claude extended thinking with a budget_tokens (≥1024, billed as output), or adaptive thinking + an effort level on current flagships (Opus 4.8, Sonnet 4.6) where the model decides how much to think. Interleaved thinking lets it reason between tool calls — pass thinking blocks back unchanged.
Use it where it pays: huge on hard, multi-step, verifiable problems; overthinking (cost + latency, no gain) on easy ones. Match the spend to the cost of a wrong answer; reasoning is a component, not a default — often a planner for cheaper agents.
Evaluate on memorization-resistant benchmarks (AIME, GPQA, ARC-AGI-2, FrontierMath) and on your tasks, scoring accuracy and compute. Treat the chain-of-thought as a useful artifact, not a faithful audit log.
Next — L257 (Computer Use & Browser Agents): put a reasoning model in a loop with a screen and a mouse — the first of the agentic-frontier lessons.