Training Data: Garbage In, Garbage Out

Last lesson we said models learn from "the internet." But the raw internet is mostly garbage — spam, duplicate pages, boilerplate, toxic content, and SEO sludge. A model can only learn from what it's shown, so if you train it on garbage, you get garbage out.

Here's the headline most people miss: the single biggest lever on a model's quality isn't the architecture — it's the data. Everyone uses transformers; what separates a great model from a mediocre one is overwhelmingly what it was fed and how that data was cleaned. Data is the secret sauce labs guard most closely.

You'll learn:

• Why garbage in = garbage out is literal here
• The data curation pipeline that turns the raw web into training data
• Why the data mixture (proportions) shapes what a model is good at
• The counterintuitive truth: quality often beats quantity
• The dark side: bias, benchmark contamination, and copyright

Recall from Language Models 101: a model learns whatever patterns help it predict the next token, with no built-in sense of what's true or trustworthy. It can't tell a careful research paper from a conspiracy forum — both are just text to imitate. So whatever is in the data, the model absorbs:

• Train it on spam and clickbait → it learns to write spam and clickbait.
• Train it on careful reasoning and clean code → it learns to reason and code.
• Train it on the toxic corners of the web → it learns toxicity.

Because the model imitates its diet, curation is not optional polish — it's the core of building a good model. The raw web, taken as-is, is not what you want a model to imitate.

Turning petabytes of raw web into good training data is a serious engineering pipeline. The standard stages:

1. Collect — gather raw sources: web crawls (Common Crawl), books, code (GitHub), Wikipedia, papers — across many languages.
2. Quality-filter — keep the good stuff, drop the junk. A common approach: score every document with a lightweight classifier and keep only the top ~10%; toss boilerplate, gibberish, and spam.
3. Safety-filter — remove toxic content, and strip personally identifiable information (PII).
4. Deduplicate — remove near-duplicate documents. The web is wildly redundant, and dedup is now standard because it both improves quality and stops the model from over-memorizing repeated text.
5. Mix — decide the proportions of each source (the data mixture) and up-sample the most valuable domains.

The output is a far smaller, far cleaner corpus than the raw input — and that is what the model actually trains on.

What you mix in — and in what proportions — directly shapes what the model becomes good at. A few well-known effects:

• More code → better reasoning. Training on lots of code improves a model's logical and step-by-step reasoning even on non-coding tasks. Code is highly structured, and that structure transfers.
• Books → long-form coherence. Long, well-edited prose teaches the model to stay coherent over many paragraphs.
• High-quality sources up-weighted. Trusted domains (Wikipedia, papers) are often up-sampled so the model sees them more.
• Language proportions → multilingual ability. A model is only as good in a language as the share of that language in its data.

The exact mixture is a closely guarded recipe — it's one of the biggest reasons two models with the same architecture and size can differ wildly in capability.

The instinct "just scrape more data" is wrong past a point. A landmark finding — Microsoft's "Textbooks Are All You Need" (the Phi models) — showed that a small model trained on carefully curated, textbook-quality data (plus some synthetic textbooks and exercises) can outperform much larger models trained on raw web scrapes.

The principle, now well established: volume without curation just amplifies failure modes, while careful curation at modest scale can match or beat uncurated data at large scale. In other words, better data often beats more data.

(This is also why synthetic data — models generating clean training data — is increasingly used. But it's delicate: a healthy mix of synthetic and real text helps, while training on purely synthetic data risks "model collapse," where the model degrades by learning only from itself. We go deeper on synthetic data in the fine-tuning section.)

If the model imitates its data, then the data's problems become the model's problems:

• Bias. Models absorb the stereotypes, skews, and blind spots present in their training data. "Garbage values in, garbage values out" — which is why you'll need guardrails and evaluation for fairness (later sections).
• Benchmark contamination. If benchmark test questions leak into the training data, the model effectively memorized the answers — inflating its scores without any real ability gain. Labs now run decontamination (removing training files that resemble test sets), but it's imperfect — which is exactly why you read benchmark numbers critically (recall that lesson).
• Copyright & legality. Much of the web is copyrighted, and whether training on it is fair use is an unsettled, actively litigated question. It's a real and growing constraint on what data can be used.

You won't curate a pretraining corpus — but this lesson hands you one of the most transferable laws in all of AI engineering:

Garbage in, garbage out applies to your data too.

The quality of the documents in your RAG index, the few-shot examples in your prompts, and the datasets you fine-tune on will dominate the quality of your results — far more than which model you pick or how clever your prompt is. The labs' obsession with data curation is the same obsession you should bring to your (much smaller) data work. It's the highest-leverage thing you control.

This lesson also explains, concretely, why your models carry biases to guard against, and why you should distrust headline benchmark numbers.

The model becomes a mirror of its data mix. Set the proportions of web, code, books, Wikipedia, and forums — toggle curation on/off — and watch the resulting model's reasoning, coherence, factuality, coding, and safety respond. Try the Reddit vs Textbook presets.

Same architecture, wildly different models — decided by the data mixture and whether it was curated. That is why labs guard their data recipe, and why better data often beats more data. The exact same law governs your RAG index, few-shot examples, and fine-tuning sets.

Predict the bias. Imagine a model pretrained only on Reddit. Predict two things it'd be unusually good at, and two it'd be bad at.

→ Good at: casual tone, internet culture, argument. Bad at: formal/academic writing, factual reliability, non-English. The lesson in one line: the model becomes a mirror of its data mix — which is exactly why frontier labs obsess over data curation, not just data quantity.

• "More data is always better." No — past a point, curation wins. Dedup and quality filtering can matter more than raw volume.
• "Benchmarks are objective truth." Contamination can inflate scores. Treat leaderboards as a starting signal, not gospel — and prefer evals on your data (later).
• "The model is neutral." It reflects its data's biases. Neutrality is something you engineer toward, not a default.
• "Data is the boring part." It's the highest-leverage part — at every scale, from pretraining down to your RAG index.

• Data is the biggest lever on model quality — bigger than architecture. The model imitates its diet, so garbage in = garbage out.
• Raw web is curated through a pipeline: collect → quality-filter (keep ~top 10%) → safety-filter → deduplicate → mix.
• The data mixture shapes capability (e.g., more code → better reasoning); it's a guarded recipe.
• Quality often beats quantity — "Textbooks Are All You Need" — and synthetic data helps in moderation (pure synthetic risks collapse).
• Data's flaws become the model's flaws: bias, benchmark contamination, copyright.
• The transferable law: garbage in, garbage out governs your data too (RAG, few-shot, fine-tuning).

Next: if data and model size both matter, how should they scale together? We meet the scaling laws that govern how models get better — and why "just make it bigger" worked.