Model Size & Scaling Laws

We've now seen two levers on model quality: data (last lesson) and size. So how do they relate — and how, exactly, do models get better? The answer is one of the most quietly profound discoveries in AI: scaling laws.

Scaling laws say that a model's performance improves predictably as you scale up compute, data, and parameters — along a smooth curve you can forecast before you train. That predictability is why labs felt safe betting hundreds of millions on "just make it bigger": they could calculate how much better the bigger model would be.

We'll keep this intuitive — no heavy math. You'll learn:

• The big idea: performance is predictable
• The three levers (parameters, data, compute) and how they balance
• The famous Chinchilla plot twist: a smaller model that won
• The "just make it bigger" era — and whether it's hitting a wall

Here's the surprise that reshaped the field. When you plot a model's loss (its prediction error — lower is better) against the compute used to train it, you don't get a messy scatter. You get a smooth, straight line going down (on a log-log plot). More compute → reliably lower loss, in a way you can extrapolate.

That means you can predict how good a bigger model will be before building it. Train a few small models, fit the line, and read off how much a 10×-bigger run will improve. This turned model-building from guesswork into something closer to engineering — and made "spend more compute, get a better model" a safe, bankable bet. It's the single biggest reason the AI investment boom happened.

Scaling laws involve three quantities that move together:

• Parameters (N) — the model's size (how many weights).
• Data (D) — how many tokens it trains on.
• Compute (C) — total training cost, roughly C ≈ N × D.

More of any one lowers loss — but they have to be balanced. You can't just keep growing the model (N) while starving it of data (D); a giant model trained on too little data is undertrained and wastes its capacity. The crucial question became: given a fixed compute budget, how should you split it between a bigger model and more data? For a few years, the field got that answer wrong.

Early models followed OpenAI's original (Kaplan) scaling laws, which leaned toward making the model bigger. GPT-3 was 175B parameters but trained on only ~300B tokens — about 1.7 tokens per parameter. It turned out to be badly undertrained: too big, too little data.

Then, in 2022, DeepMind's Chinchilla paper dropped a bombshell. At the same compute budget as the 280B-parameter Gopher, they trained a model with only 70B parameters but 4× more data — and the 70B Chinchilla beat the 280B Gopher (and GPT-3, and others).

The takeaway rewired the whole field: don't just scale the model — scale parameters and data together. The compute-optimal ratio is roughly ~20 tokens per parameter (vs GPT-3's 1.7). It's not "biggest model wins"; it's "best-balanced model wins." This is exactly why modern models train on trillions of tokens relative to their size (recall: Llama 3, 15T tokens).

For years, scaling laws gave the field a deceptively simple recipe: scale it up. GPT-2 → GPT-3 → GPT-4 was, in large part, more compute, more data, more parameters → reliably better model. No fundamentally new idea was strictly required — just bigger, well-balanced runs.

This is why the foundations you learned in Section 2 were so important: the Transformer's parallelism is what made this scaling possible in the first place. Scaling laws + a parallelizable architecture = the engine of the entire LLM boom.

Pure scaling can't run forever, and around 2026 two pressures are biting:

• The data wall. Chinchilla says you need lots of data — but there's a finite amount of high-quality human text. Estimates suggest frontier models will exhaust the high-quality public web roughly between 2026 and 2032. You can't get 20 tokens per parameter if the tokens don't exist.
• Diminishing returns. The scaling line is predictable but also shallow — each 10× of compute buys a smaller and smaller capability gain, while costing 10× more.

So the frontier is shifting beyond brute pretraining scale toward: better data (last lesson), synthetic data, and test-time compute — letting models "think longer" at inference (the reasoning models we'll meet soon) instead of just being bigger. Scaling laws aren't dead, but they're no longer the only lever.

You'll never run a scaling experiment, but this shapes decisions you will make:

• Bigger is usually better — but not always. A well-trained smaller model can beat a poorly-trained bigger one (Chinchilla). So when choosing a model, test on your task rather than assuming the biggest wins (recall the model-selection lesson).
• That's why model families come in sizes (small / medium / large). You'll pick along the latency-cost-quality tradeoff — often a smaller model is the right call.
• Progress is changing shape. As pretraining scaling slows, gains increasingly come from data quality, post-training, and reasoning (test-time compute) — which is good news for application builders, because those levers are closer to you.

Do the Chinchilla math. The compute-optimal rule of thumb is ~20 training tokens per parameter.

• A 10B-parameter model → how many training tokens is 'compute-optimal'? → ~200B tokens.
• You have a fixed compute budget and a 70B model trained on only 200B tokens. Compute-optimal, or under-trained? → under-trained (it 'wants' ~1.4T tokens).

This is why 'just make it bigger' gave way to 'balance size and data.'

• "Biggest model = best." No — compute-optimal wins. An undertrained giant loses to a well-fed smaller model (Chinchilla 70B > Gopher 280B).
• "Scaling is infinite." It isn't — the data wall and diminishing returns are real, which is why the field is diversifying its levers.
• "Scaling laws guarantee AGI." They predict loss, not capability ceilings or 'understanding.' Lower loss ≠ guaranteed new abilities.
• "You should always pick the largest model." Often the smaller model is the smart, cheaper, faster choice — measure, don't assume.

• Scaling laws: a model's loss falls predictably as you scale compute, data, and parameters — you can forecast a bigger model's quality before training it.
• The three levers (N params, D data, C ≈ N×D compute) must be balanced.
• Chinchilla: compute-optimal is ~20 tokens per parameter — a 70B model beat a 280B one at equal compute. Balance beats brute size.
• The "just make it bigger" era worked because of scaling laws + the Transformer's parallelism — but the data wall and diminishing returns are now pushing the frontier toward better data and reasoning (test-time compute).
• For you: bigger is usually better, but not always — test on your task and choose by latency/cost/quality.

Next: with the how-they're-built picture complete, let's meet the actual players — the model families (GPT, Claude, Gemini, Llama, Qwen, Mistral, DeepSeek) you'll build with.