Multi-Head Attention & Positional Encoding

Introduction

Our self-attention picture is powerful but has two gaps, and this lesson plugs both:

A single attention can only track one kind of relationship at a time — but language has many at once (grammar, coreference, topic…).
Attention as we described it is blind to word order — yet "dog bites man" and "man bites dog" mean very different things.

The fixes are multi-head attention (run many attentions in parallel) and positional encoding (tell the model where each token sits). Add these two and you have the complete attention machinery used in real LLMs.

You'll learn:

Why one attention head isn't enough — and what multi-head buys you
How heads act as a committee of specialists
Why self-attention is order-blind (and why that's a disaster)
How positional encoding restores order — sinusoidal, learned, and modern RoPE

Part 1 — The Problem With a Single Head

Last lesson, one attention learned a single, beautiful skill: resolving "it" → "animal." But a sentence is dense with simultaneous relationships the model needs to track:

Coreference — which noun does this pronoun refer to?
Syntax — which verb goes with which subject?
Local grammar — which adjective modifies which noun?
Long-range topic — what's this whole passage about?

One attention head has to compromise — it can specialize in one of these, but not all. Forcing a single head to juggle everything makes it mediocre at each. The fix is obvious once you see it: don't use one head — use many.

Multi-Head Attention = a Committee of Specialists

Multi-head attention runs several attention heads in parallel — modern models use dozens (e.g., 12, 32, 64). Each head gets its own learned Query/Key/Value projections, so each is free to learn a different lens on the sentence:

Head 1 might track pronoun resolution ("it" → "animal").
Head 2 might track subject–verb agreement.
Head 3 might attend to the adjective → noun it describes.
Head 4 might capture long-range topic.
…and so on.

Think of it as a committee of specialists: each member reads the same sentence through a different lens, then they pool their findings. After all heads run, their outputs are concatenated and projected back together into one vector per token — now enriched with many kinds of context at once.

A diagram of multi-head attention. One input sentence is fed into several parallel attention heads, each specializing in a different relationship: Head 1 (pronouns) attends 'it' to 'animal'; Head 2 (subject-verb) links a verb to its subject; Head 3 (adjective-noun) links a modifier to its noun; Head 4 (long-range topic) spreads attention broadly. The heads' outputs are then concatenated and projected into a single combined representation, with a note that real models use dozens of heads.

How the Heads Split the Work

A common worry: "isn't running 32 attentions 32× more expensive?" No — the model's vector dimension is split across the heads. If the model works in 4,096 dimensions with 32 heads, each head works in a smaller 128-dimensional slice. Same total compute, just divided into parallel specialists.

In pseudocode, multi-head attention is simply "run attention H times, then combine":

# Multi-head attention = H single-head attentions in parallel, then combined
heads = [
    self_attention(Q_h, K_h, V_h)   # each head: its OWN learned Q/K/V projections
    for h in range(H)               # e.g. H = 32 heads
]
out = concat(heads) @ W_o           # stitch the heads back together, then project
# Each head specialized in a different relationship; `out` carries them all.

That's the entire idea. One head was a soloist; multi-head is an orchestra — and the model learns what each section should play.

Part 2 — Attention Is Order-Blind

Now the second gap, and it's a big one. Look closely at how attention works: each token compares its Query to every Key and blends Values. Nothing in that process refers to position. Attention treats the input as an unordered bag of tokens — a set, not a sequence.

That means, to pure self-attention, these two sentences are identical:

"dog bites man"     →  { dog, bites, man }
"man bites dog"     →  { dog, bites, man }   ← same set!

Same tokens, same set — yet opposite meaning. If the model can't tell them apart, it can't understand language at all. We must inject word order.

The Fix: Position Stamps

The solution is to give every token a position stamp — extra information about where it sits — before attention runs. "dog" at position 1 becomes distinguishable from "dog" at position 3. There are three approaches you'll hear about:

Sinusoidal (the original). Add fixed sine/cosine wave patterns of different frequencies to each embedding — a unique "fingerprint" per position that also encodes relative distance and extends to lengths never seen in training.
Learned positional embeddings. Just learn a vector for each position (GPT-2 did this). Simple, but caps the maximum length.
RoPE — Rotary Position Embeddings (the modern standard). Used by Llama and most 2026 models. Instead of adding a position vector, RoPE rotates each token's Query and Key by an angle proportional to its position. The clever part: the difference in rotation between two tokens encodes their relative distance — which helps a lot with long context.

You don't need to memorize the math. The takeaway: every token gets stamped with its position, and modern LLMs use RoPE to do it.

An illustration of why positional encoding is needed. On the left, the sentences 'dog bites man' and 'man bites dog' both reduce to the same unordered set {dog, bites, man}, so pure self-attention cannot tell them apart (marked with a red X). On the right, each token gets a position stamp (pos 1, 2, 3) added to its embedding, so the two sentences become distinguishable (green check), preserving their opposite meanings. A note says modern LLMs use RoPE.

See It: Order Matters

Prove order-blindness to yourself. Swap the word tiles, then toggle position stamps. With stamps OFF, the model receives an unordered set — every arrangement is the same input. With stamps ON, each token carries where it sits, and “dog bites man” finally differs from “man bites dog.”

Interactive: Order Matters. The user reorders three word tiles — dog, bites, man — by clicking two tiles to swap them, and toggles position stamps on or off. With stamps off, a panel shows that the model receives an unordered set written as a bag of tokens in braces, so the sentences dog bites man and man bites dog are the same input and the subject and object are lost; reordering changes nothing. With stamps on, each tile gains a superscript position number and the panel shows the model receives an ordered sequence, where position one is the subject and position three is the object, so the meaning flips when the user reorders the tiles and the model can tell the two sentences apart. A status line echoes the current order and labels it distinguishable or ambiguous. The interactive makes concrete the lesson's core point that pure self-attention is order-blind and behaves like a bag-of-words, and that real language models fix this by stamping every token with its position, with modern models using RoPE, so the true input is always meaning plus position.

That is exactly why Transformers need positional encoding: without it, attention is a fancy bag-of-words. Every token is stamped with its position before attention runs — modern LLMs use RoPE — so the true input is always meaning + position.

The Complete Input

Now the architecture lesson's first steps make full sense. Before the Transformer blocks run, each token's representation is:

token embedding  (what the token means)
      +
positional info  (where the token sits)

That combined vector flows into the stack, and inside every block, multi-head self-attention does the mixing. So the real recipe is: meaning + position → many parallel attention heads → feed-forward → repeat. You now understand every box in that diagram.

🧪 Try It Yourself: Different Heads, Different Focus

Multi-head attention runs several attention patterns in parallel, each learning to track something different. Flip between Head 1 and Head 2 below and click “it”: one head resolves coreference (it → animal), another tracks adjacency (nearby words). Each head is a different lens on the same sentence — together they capture far more than one could.

Multi-head attention — switch heads, see different focus.

Mental-Model Corrections

Heads aren't redundant copies. Each has its own learned projections and specializes; the model discovers useful divisions of labor (and yes, some heads end up doing little — that's normal).
Multi-head isn't more expensive. The dimension is split across heads; total compute is roughly the same as one big head.
Positional encoding isn't something you manage at inference. It's baked into the model. You don't add position stamps in your prompt — the model does it internally.
You don't need the RoPE math. Know that modern models encode position (and that RoPE encodes relative position, which helps long context). Names, not derivations.
Order-blindness was the whole reason transformers need positional encoding — without it, a transformer is just a fancy bag-of-words.

Key Takeaways

Multi-head attention runs many attention heads in parallel, each a specialist in a different relationship (coreference, syntax, topic…). Their outputs are concatenated and projected.
It's not more expensive — the model dimension is split across heads.
Pure self-attention is order-blind: it sees a set of tokens, so "dog bites man" = "man bites dog" without help.
Positional encoding stamps each token with its position. Approaches: sinusoidal, learned, and the modern standard RoPE (encodes relative position, great for long context).
The true input to a Transformer is token meaning + position, mixed by multi-head attention in every block.

Next: we've built the Transformer piece by piece. Now we ask why it crushed the previous champion — Why Transformers Won (vs RNNs/LSTMs) — and what that means for everything you build.