Tokenization: How Text Becomes Numbers (BPE)

Introduction

Last lesson we used "token ≈ word" as a training-wheels simplification. Time to take the wheels off.

Models don't process words, and they don't process letters. They process tokens — integer IDs for chunks of text — and the very first thing that happens to your prompt is tokenization: text gets chopped into tokens and turned into numbers before the model sees anything.

This sounds like a boring plumbing detail. It isn't. Tokenization explains a surprising amount of real model behavior — why ChatGPT can't reliably count the R's in "strawberry", why your API bill is shaped the way it is, and why some languages cost 2–3× more than English to process.

You'll learn:

Why models use subword tokens (not words or characters)
What a token actually is — and what a vocabulary is
Byte-Pair Encoding (BPE) — how that vocabulary is built
How to tokenize text yourself with code
The real-world consequences (the strawberry problem, cost, multilingual)

Why Not Just Words or Characters?

You might expect the model to split on words, or on individual characters. Both extremes break:

Whole words → the vocabulary explodes. Every word, plural, tense, typo, name, and new coinage ("rizz", "GPT-4o") needs its own entry — millions of them — and you still can't handle a word you've never seen.
Single characters → the vocabulary is tiny, but every sentence becomes a very long sequence. The model burns its limited capacity and context window modeling spelling instead of meaning.

Subword tokenization is the sweet spot in between. Common words become a single token; rare or novel words get split into reusable pieces; and anything can be represented by falling back to smaller fragments (down to bytes). You get a manageable, fixed vocabulary (today typically ~100,000–200,000 tokens) that can encode any text. The dominant method for building it is Byte-Pair Encoding.

What a Token Actually Is

A token is just an entry in the model's fixed vocabulary — a chunk of text paired with an integer ID. Tokenization maps your text to the sequence of IDs, and the model only ever works with those numbers.

"The cat sat."
   → tokens:  ["The", " cat", " sat", "."]
   → IDs:     [976,   9059,   10139,  13]

Two things to notice immediately:

The leading space is part of the token. " cat" (with a space) is a different token from "cat". (Remember the logprobs demo last lesson showing both 'Paris' and ' Paris'? Same reason.)
One token is not one word. A rough, useful heuristic for English: 1 token ≈ 0.75 words ≈ 4 characters (so ~1.33 tokens per word). You'll use this constantly to estimate cost and context budget.

Byte-Pair Encoding (BPE) — How the Vocabulary Is Built

Where does the vocabulary come from? Mostly from Byte-Pair Encoding — an old compression idea (Philip Gage, 1994) adapted for language models. The algorithm is delightfully simple:

Start with the smallest units — individual characters (or bytes).
Count every adjacent pair across a huge training corpus.
Merge the most frequent pair into a new token, and add it to the vocabulary.
Repeat thousands of times — until the vocabulary reaches the target size.

Frequent letter-sequences ("th", "ing", "tion", common whole words) get merged early and become single tokens; rare strings never merge and stay fragmented. That's exactly why common words are one token and unusual ones are split.

An illustration of Byte-Pair Encoding building a vocabulary. Starting from individual characters of a tiny corpus (low, lower, newest, widest), the algorithm repeatedly merges the most frequent adjacent pair: step 1 merges 'e' + 's' into 'es', step 2 merges 'es' + 't' into 'est', step 3 merges 'l' + 'o' into 'lo' then 'lo' + 'w' into 'low'. Each step adds a new token to the vocabulary, with a note that this repeats until the vocabulary reaches ~100k-200k tokens.

Tokens in Practice

You can tokenize text yourself in seconds. OpenAI's tokenizer library, tiktoken, runs locally (no API call, free). Modern OpenAI models use the o200k_base encoding (a ~200k-token vocabulary); GPT-4/3.5 used cl100k_base.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")   # GPT-4o / 4.1-era vocabulary

ids = enc.encode("Tokenization isn't magic.")
print(ids)                                   # -> [2380, 2065, ... ]  (integer IDs)
print([enc.decode([i]) for i in ids])        # the actual token strings
print(len(ids), "tokens")

# The strawberry problem, made visible:
print([enc.decode([i]) for i in enc.encode("strawberry")])
# -> ['st', 'raw', 'berry']     (3 tokens — the model never sees the letters)

Note that Anthropic's Claude uses its own tokenizer (not tiktoken), so exact token counts differ between providers — but the concept is identical. For estimating Claude usage, rely on the usage field the API returns (you saw it in the API-anatomy lesson) or Anthropic's token-counting endpoint.

Why This Matters: The Strawberry Problem & Friends

Tokenization isn't trivia — it directly causes behavior you'll hit in production.

1. The famous "strawberry" failure. Ask a model "how many R's are in 'strawberry'?" and it often says 2 (the answer is 3). Why? Because the model never sees the letters — it sees the tokens ['st', 'raw', 'berry']. To count R's it would have to know each token's letter contents (st = 0, raw = 1, berry = 2) and sum them. It's not "dumb"; it's blind to spelling by construction.

An explanation of why a model miscounts the R's in 'strawberry'. The word 'strawberry' is shown splitting into three colored token chips: 'st', 'raw', and 'berry'. Under each chip is its count of the letter R: st has 0, raw has 1, berry has 2, summing to 3. A caption explains the model sees tokens, not letters, so it never directly observes the individual R's and commonly answers 2.

2. Cost & context are billed in tokens. Your bill and your context window are measured in tokens, not words (recall the API usage field). The 0.75-words-per-token heuristic is how you estimate both.

3. Some languages cost far more. Tokenizers are trained mostly on English, so English is compact (~1.3 tokens/word) while many other languages — and lots of code or unusual formatting — fragment into many more tokens. The same meaning can cost 2–3× more in tokens (and money, and latency) in another language.

4. Rare words and typos fragment. An unusual name or a typo splits into many small tokens, which can confuse the model and waste context.

🧪 Try It Yourself: The Tokenizer

Type anything below and watch it shatter into tokens in real time:

Try a long, rare word like antidisestablishmentarianism → it splits into many sub-word pieces (the model has no single token for it).
Try GPT-4 costs $0.03 🤖 → see how numbers, symbols, and emoji tokenize differently from words.
Watch the chars-per-token number hover near ~4 — the rule of thumb you'll use to estimate tokens (and cost) forever.

Tokenizer playground — type text, watch it split into tokens.

Mental-Model Corrections

Don't ask models to do character-level work blindly. Counting letters, reversing strings, strict character manipulation — these fight the tokenizer. If you must, give the model a tool (e.g., run code) instead of trusting it to "see" letters.
Estimate, then measure. Use ~0.75 words/token to plan, but read the real usage numbers from the API for anything that matters (cost, context limits).
Token counts are provider-specific. Don't assume OpenAI's count equals Claude's — different tokenizers, different numbers.
"token ≈ word" is fine for intuition, but now you know the truth underneath — and why spelling-sensitive tasks misbehave.

Key Takeaways

Models read tokens — integer IDs for subword chunks — not words or letters. Tokenization is the first step on every prompt.
Subwords balance the word-vs-character tradeoff; the vocabulary (~100k–200k tokens) is built with Byte-Pair Encoding: repeatedly merge the most frequent adjacent pair.
Rule of thumb: 1 token ≈ 0.75 words ≈ 4 English characters; leading spaces are part of tokens; counts differ per provider.
Tokenization explains real behavior: the strawberry letter-counting failure, token-based cost/context, and why non-English text costs more.
For character-level tasks, reach for a tool, not the model's "eyes."

Next: tokens are still just IDs with no inherent meaning. We give them meaning with embeddings — turning tokens into vectors where similar things sit close together.