The Transformer Architecture — The Big Picture

Introduction

Meet the engine inside every modern LLM: the Transformer (it's literally the "T" in GPT — Generative Pre-trained Transformer). It's the architecture that, after years of slow progress, suddenly made models that can write, code, and reason.

We are not going to derive equations here. Our goal is the mental model — the shape of the machine — so that when we open up attention in the next lesson, you already know where it fits. Get the big picture and the details have somewhere to land.

The one-sentence version: a Transformer passes your tokens through a tall stack of identical blocks that progressively refine each token's vector — mixing in context from the other tokens — until the final vector is rich enough to predict the next token.

You'll learn:

The big idea: a refinement stack
The end-to-end flow, from text to next-token probabilities
What lives inside one block (attention + feed-forward + the glue)
Why models are deep (many layers), and why that matters
Why this design was revolutionary

The Big Idea: A Refinement Stack

Remember where we left off: tokens become embeddings — vectors with meaning, but each considered in isolation. The embedding for "bank" is the same whether you're talking about a river or a loan. That's not enough.

The Transformer's whole job is to add context. Think of it as an assembly line: a token vector enters, and at each station (each layer) it gets refined by looking at the other tokens around it. By the end of the line, "bank" in "sat by the river bank" has absorbed "river" and become a context-aware vector that means the right thing.

So a Transformer is a stack of identical layers that take in vectors and put out better vectors — same shape in, same shape out, just richer in meaning each time. The final vector for the last token is what we use to predict what comes next.

The End-to-End Flow

Here is the complete journey of your prompt, start to finish. Most of these steps you've already met:

Tokenize → text becomes token IDs (lesson: Tokenization)
Embed + position → each ID becomes a vector, plus a signal for where it sits in the sequence (embeddings + positional info — more on position later)
N Transformer blocks → the refinement stack does its work
Final linear + softmax → the last token's vector becomes a probability distribution over the whole vocabulary (lesson: Next-Token Prediction)

That's the entire pipeline. Everything novel happens in step 3 — let's open it up.

A top-to-bottom diagram of the Transformer pipeline. Input text 'The capital of France is' is tokenized into token IDs, then embedded into vectors with positional information. The vectors pass through a Transformer block repeated N times (e.g. 32 layers); each block contains a Self-Attention sub-layer ('mix in context — each token looks back at the others', with Add & Norm) followed by a Feed-Forward sub-layer ('process each token — think', with Add & Norm). After the stack, a final linear layer plus softmax produces a next-token probability distribution where 'Paris' is about 91%.

Inside a Transformer Block

Every block in the stack is identical, and it has just two main parts plus some glue:

① Self-Attention — the mixing step. Each token looks at the other tokens and pulls in whatever context is relevant ("river" informs "bank"). This is how words relate, and it's the heart of the Transformer. We devote the entire next lesson to it. (In LLMs it's causal/masked: a token may only look at itself and earlier tokens — never the future — which is exactly what next-token prediction needs.)
② Feed-Forward Network — the thinking step. After mixing, each token's vector is passed (independently) through a small neural network — two layers with a nonlinearity between — that transforms it further. If attention decides what to look at, the feed-forward step decides what to make of it.

And the glue that makes deep stacks trainable:

Residual (skip) connections — each sub-layer adds its output back onto its input, so information and gradients flow cleanly through dozens of layers (nothing gets lost).
Layer normalization — rescales the numbers at each step to keep them stable. You'll see these together as "Add & Norm."

That's the whole block: attention → feed-forward, each wrapped in add & norm. Stack it N times and you have a Transformer.

Why a Stack? Depth = Capability

One block adds a little context. The magic is in stacking many — small models have ~12 layers, large ones 32, 80, or more.

Why depth helps: layers learn to build meaning hierarchically. Early layers tend to capture surface patterns (grammar, which words go together nearby); middle and later layers capture increasingly abstract structure (who a pronoun refers to, sentiment, logical relationships). It's loosely like image models that build from edges → shapes → objects — here it's characters → grammar → meaning → reasoning.

This is a big part of why scale works: more layers (and wider ones) give the model more capacity to refine meaning — which is exactly the lever the labs pulled to get from GPT-2 to today's frontier models.

The Shape of a Forward Pass

If you're the kind of person who understands things better in code, here's the entire forward pass in pseudocode. Notice how short it is — it really is just a loop over blocks:

# Pseudocode — the skeleton of every decoder-only LLM
x = embed(token_ids) + positional(token_ids)   # vectors, with position

for block in blocks:               # e.g. 32 identical layers
    x = x + self_attention(norm(x))   # ① mix context (look back only)
    x = x + feed_forward(norm(x))     # ② think, per token
    #   ^residual (+x)   ^layer norm

logits = final_linear(norm(x))     # vector -> score per vocabulary token
next_token_probs = softmax(logits[-1])   # distribution for the NEXT token

That for loop is the Transformer. Everything else in this section — attention, positional encoding, multi-head — is detail about what goes inside self_attention and embed. And in lesson 8 of this section, you'll actually build this for a tiny GPT, line by line.

Why It Was Revolutionary

Before Transformers (2017's "Attention Is All You Need"), the best language models were RNNs that read text one word at a time, passing along a single running summary. That was slow (can't parallelize) and forgetful over long distances (the start of a paragraph fades by the end).

The Transformer changed two things:

Parallelism — it processes all tokens at once, so it trains efficiently on huge data and hardware (the key to scaling).
Direct long-range connections — via attention, any token can look directly at any earlier token, no matter how far back, so context isn't lost.

Those two properties are why it was possible to scale to today's models. (We'll do the full Transformers-vs-RNNs comparison in a later lesson — for now, just hold onto parallel and direct attention.)

See It: Trace a Token

Follow one token up the stack. Pick a context for the ambiguous word “bank,” then step it through Embed → Block 1 → Block 2 → Block 3 → Final softmax, and watch its meaning sharpen from a coin-flip into the right sense — purely from context.

Interactive: Trace a Token through the Transformer. The user picks one of two contexts for the ambiguous word bank — the sentence she sat by the river bank, or she opened a bank account today — and then steps that token up the refinement stack through five stages: embed plus position, Block 1, Block 2, Block 3, and the final softmax. A live meaning meter shows the two competing senses of bank, a river's edge versus a financial institution, starting at a fifty-fifty coin flip when the token is an isolated embedding and sharpening toward the correct sense as depth increases, reaching roughly ninety-seven percent confidence by the final layer. Each stage is annotated to explain that attention lets the token look back and pull in nearby context so the tokens communicate, the feed-forward network refines each token's vector per token, deeper layers capture increasingly abstract structure, and the final refined vector becomes a probability distribution over the vocabulary. The interactive makes concrete the lesson's core ideas that a Transformer is a stack of identical blocks producing the same shape in and a richer vector out, that attention is where tokens communicate while the feed-forward thinks, and that depth equals capability because more layers turn ambiguity into resolved meaning.

Same word, same starting vector — only the context differs, and the stack resolves it. That is the whole architecture in motion: identical blocks where attention lets tokens communicate and depth turns ambiguity into meaning.

🧪 Try It Yourself

Trace a token. Mentally follow a word through the transformer: embedding → (attention → feed-forward) ×N → unembedding → next-token.

Question: which step is the one that lets words look at each other (so "it" can figure out it means "the cat")? → attention. The feed-forward layers think about each token; attention is where tokens communicate. Hold that split — it's the heart of the architecture.

Mental-Model Corrections

"Transformer = attention." Attention is the star, but a block is attention + feed-forward, wrapped in residual + layer-norm, and stacked deep. All the parts matter.
"It understands like a human." It refines vectors with learned statistics — extraordinarily effective, but it's matrix math, not comprehension. Keep that honesty.
Depth isn't memory across calls. A 96-layer model still forgets everything between API calls (statelessness from the API lesson). Layers add context within one forward pass, not memory across requests.
Same shape in, same shape out. Each layer takes token vectors and returns token vectors of the same size — it refines, it doesn't summarize down to one thing (until the very end).

Key Takeaways

A Transformer is a stack of identical blocks that progressively refine token vectors by adding context — turning isolated embeddings into context-aware representations.
The flow: tokenize → embed + position → N blocks → final linear + softmax → next-token probabilities.
Each block = Self-Attention (mix in context — look back) + Feed-Forward (think, per token), each wrapped in Add & Norm (residual + layer normalization).
Depth = capability: stacking many layers builds meaning hierarchically; it's a core reason scale works.
It was revolutionary for parallelism + direct long-range attention — the foundations of scaling.

Next: we open the most important box of all — Self-Attention — and answer, intuitively, how does a token decide which other tokens to pay attention to?