Pretraining: Learning from the Internet

You just built a tiny GPT and trained it on a single small text file. Pretraining is that exact process — self-supervised next-token prediction — turned up to a scale that's genuinely hard to picture: trillions of tokens, thousands of GPUs, months of training, and millions of dollars.

This is the step that creates a foundation model (the "foundation" in the name). It's the most expensive, most important, and least-often-done part of the whole pipeline — and the reason you, as an AI engineer, get to rent frontier intelligence instead of building it.

You'll learn:

• Why pretraining is just your tiny GPT, massively scaled
• The almost unimaginable scale (data, compute, cost)
• What "learning from the internet" actually means
• What pretraining produces — a base model (and why it's not yet a chatbot)
• Why all of this matters even though you'll never do it yourself

Here's the reassuring part: you already understand the algorithm. Pretraining a frontier model uses the same objective you just coded — predict the next token, compare to the real one, nudge the weights, repeat (self-supervised, no human labels). Nothing about the method is new.

What changes is the scale of three dials:

• Data — from one file to a huge slice of everything humans have written.
• Model — from 4 layers to dozens or hundreds; from thousands of parameters to billions or trillions.
• Compute — from your laptop for a few minutes to thousands of GPUs for months.

That's it. The leap from your toy to GPT-class isn't a different machine — it's the same machine with the dials turned to 11. Which is exactly why understanding the toy means you understand the real thing.

Numbers make it concrete. A modern foundation model is trained on tens of trillions of tokens — for example, Llama 3 used ~15 trillion, and newer models push toward 30T+ (increasingly including images and video, not just text). Models have billions to trillions of parameters, trained on thousands of high-end GPUs (H100/B200-class) running for weeks to months.

The price tag is staggering and mostly estimated (labs rarely publish exact figures): widely-cited numbers run from a few million dollars for earlier models to well over $100 million for frontier training runs — before you count the research, the data work, and the failed attempts.

The "text file" for a foundation model is a massive, curated mixture of human writing: web pages (filtered crawls of the internet), books, code (e.g., GitHub), Wikipedia, scientific papers, and more — across many languages.

Now recall the deep idea from Language Models 101: predicting the next token well forces the model to learn whatever underlies the text. Do that across a huge fraction of everything ever written, and the model implicitly absorbs grammar, facts, reasoning patterns, coding ability, and world knowledge — not because anyone programmed them in, but because they're useful for predicting the next token. That emergent competence, at this scale, is what we call a foundation model. (How that data is filtered and curated is so important it gets its own lesson next — here, just hold the picture: predict-the-next-token, over the internet.)

Here's a crucial surprise: the direct output of pretraining is not a helpful chatbot. It's a base model — an extraordinary next-token predictor and continuer of text, but one that doesn't yet know it's supposed to assist you.

Ask a raw base model "What is the capital of France?" and it might not answer — it might continue the pattern with more questions:

Prompt:  What is the capital of France?
Base:    What is the capital of Germany? What is the capital of Spain? …

Why? Because in its training data, a line like that often appears in a list of quiz questions — so the statistically "likely continuation" is more questions, not an answer. The base model is doing its job perfectly; its job just isn't "be helpful."

To turn this knowledgeable-but-unhelpful base model into the assistant you actually talk to (Claude, ChatGPT), it needs a second phase — post-training — which we cover in its own section. (And the base-vs-instruct distinction gets its own lesson later in this section.)

Zoom out and a foundation model is made in two phases:

1. Pretraining (this lesson) — hugely expensive, done once, on the whole internet. Produces a base model packed with knowledge and capability.
2. Post-training (a later section) — far cheaper, makes the model helpful, safe, and instruction-following (SFT, RLHF/DPO).

A useful intuition: pretraining gives the model its knowledge and raw ability; post-training gives it its manners and helpfulness. Almost all the cost and world-knowledge live in pretraining; almost all the "assistant personality" lives in post-training.

You will almost certainly never pretrain a model — it's a frontier lab's job, costing millions. But understanding it explains several things you'll deal with constantly:

• The knowledge cutoff. Pretraining happens once, at a point in time, and freezes the model's knowledge there. That's why models don't know recent events — and a core reason RAG exists (recall: retrieve fresh info at query time).
• Why models "know" so much. The internet was effectively compressed into the weights during pretraining.
• Why frontier models are so few. Only a handful of organizations can afford a $100M+ training run — which is exactly why you rent models (build-vs-buy) rather than build them.
• Why hallucination happens. A base model predicts plausible continuations from compressed memory — not verified facts.

Pretraining = predict the next token across the internet. Sort each capability: did it emerge in pretraining (knowledge), get added in post-training (manners), or does it need RAG/tools (live or private info the frozen weights never saw)?

Three different problems, three different fixes: pretraining compresses the internet into frozen weights (knowledge), post-training adds the assistant's manners, and RAG/tools supply the live or private facts the weights never saw. That split explains the knowledge cutoff — and why RAG exists.

What did pretraining actually teach? Pretraining = predict the next token across the internet. For each, predict whether the model learns it here or later (post-training):

1. English grammar → ? 2. Facts about the world → ? 3. To be a helpful, safe assistant → ?

→ 1 and 2: yes, in pretraining (knowledge + language emerge from prediction). 3: no — helpfulness/safety come from post-training. That's why a raw base model knows everything but won't follow your instructions.

• The model is not looking things up on the internet. During inference it has no connection to anything — the internet was compressed into its frozen weights at training time. That's why it has a knowledge cutoff and can hallucinate. (Giving it live access is what tools and RAG do — later.)
• Pretraining doesn't make a chatbot. It makes a base model. The helpful assistant comes from post-training.
• More data/parameters isn't automatically better in a simple way — there's a science to how they should scale together. That's the scaling laws lesson, coming up.
• You won't pretrain. Internalize the concept; rent the result.

• Pretraining is the same next-token, self-supervised objective as your tiny GPT — scaled to trillions of tokens, billions–trillions of parameters, thousands of GPUs, and $millions–$100M+.
• "Learning from the internet" means absorbing grammar, facts, reasoning, and code as emergent byproducts of predicting the next token over a huge corpus.
• The output is a base model: brilliant at continuing text, but not yet a helpful assistant — that requires post-training.
• Pretraining = knowledge & ability; post-training = helpfulness & manners.
• It explains the knowledge cutoff, why RAG exists, why there are so few frontier models, and why you rent rather than build.

Next: the single biggest lever on a model's quality isn't the architecture — it's the data. We look at Training Data: Garbage In, Garbage Out.