Self-Attention, Intuitively

Introduction

This is the big one. Self-attention is the mechanism that made Transformers work — the "mixing step" inside every block — and it's the concept most people find slippery. So we're going to build it up purely from intuition, no scary math, until it feels obvious.

Here's the puzzle it solves. Read this sentence:

"The animal didn't cross the street because it was too tired."

What does "it" refer to — the animal, or the street? You knew instantly: the animal. Your brain, reading "it," automatically looked back to find what it points to. Self-attention is how a model does that same thing — letting each token look at the others and pull in the context it needs.

You'll learn:

Why a token's meaning is a weighted blend of the other tokens
The Query / Key / Value idea — as a soft database lookup
A worked example of "it" finding "animal"
Why it only looks backward in an LLM
The whole thing in ~3 lines of code

The Core Idea: A Weighted Blend

Recall from the embeddings lesson: each token starts as a vector, but in isolation. "it" has no idea it means "animal" yet. Attention fixes this by replacing each token's vector with a weighted blend of information from the other tokens — paying more attention to the relevant ones.

For "it," the blend should be mostly "animal," with small contributions from the rest:

new vector for "it"  ≈  0.70 · (info from "animal")
                     +  0.10 · (info from "street")
                     +  0.05 · (info from "tired")
                     +  … (small bits of everything else)

So the entire job of attention boils down to one question, asked for every token: "how much should I attend to each of the other tokens?" Get those weights right, blend accordingly, and "it" walks out of the block meaning "the animal." The clever part is how the model computes those weights — and that's where Query, Key, and Value come in.

Query, Key, Value — A Soft Database Lookup

The mechanism is best understood as a soft database lookup. In a normal database you have a query, you match it against keys, and you retrieve the matching values. Attention does exactly this — just softly (blending matches instead of picking one).

From each token's vector, the model produces three new vectors (via three weight matrices learned during training):

Query (Q) — "what am I looking for?" ("it" asks: what singular noun could be the subject of 'was too tired'?")
Key (K) — "what do I offer? what am I about?" — like a label other tokens advertise.
Value (V) — "the actual information I'll hand over" if you attend to me.

The recipe for one token:

Take its Query and compare it against every token's Key → a relevance score for each.
Softmax those scores → attention weights that sum to 1.
Take the weighted sum of all the Values → the token's new, context-aware vector.

That's it. A query searches the keys; you retrieve a blend of the values, weighted by how well each key matched.

A three-step diagram of self-attention as a soft lookup. Step 1: the token 'it' forms a Query ('what singular noun is the subject of was too tired?'). Step 2: the Query is compared against the Key of every other token, producing attention weights shown as bars — 'animal' gets the largest weight (0.70), with small weights for 'street' (0.10), 'tired' (0.05) and others. Step 3: the Values of all tokens are blended by those weights to produce a new context-aware vector for 'it', dominated by the Value of 'animal'.

Worked Example: How "it" Finds "animal"

Let's run our sentence through it. As the model processes "it", it forms a Query that (roughly) asks "what singular, non-human noun could be the subject of 'was too tired'?" It compares that Query against the Key of every word:

"animal" — its Key is a great match → high attention weight.
"street" — a noun, but "streets" don't get tired → low weight.
"the", "cross", "because" — barely relevant → tiny weights.

The model then blends the Values by those weights, so "it"'s new vector is dominated by the Value of "animal" — its rich meaning gets "baked into" "it." From this layer on, the model effectively knows that "it" = "the animal." Pronoun resolved, by geometry.

An attention visualization for the sentence 'The animal didn't cross the street because it was too tired.' The word 'it' is highlighted, and connection lines (and weight bars under each word) show how much 'it' attends to each earlier word. The line to 'animal' is by far the thickest (highest weight), with thin lines to other words. A note says 'it' can only attend to itself and earlier words (causal).

It Only Looks Backward (Causal Attention)

One important constraint, carried over from the last lesson: in an LLM, a token may only attend to itself and the tokens before it — never the future. This is causal (masked) attention.

Why? Because the model's whole job is next-token prediction. When it's predicting the word after "because," the words "it was too tired" don't exist yet — so it mustn't be allowed to peek at them. During training we enforce this with a mask that zeroes out attention to future positions. So "it" can attend to "animal" (which came earlier) but not to "tired" (which comes later). Every token sees only its own past.

Seeing It All at Once: The Attention Map

So far we've followed a single token ("it"). But every token attends at the same time — and the standard way to see all of it at once is an attention map: a grid where each row is a token doing the attending and each column is a token being attended to. Darker = more attention.

This visual shows up everywhere in AI papers and tools, so get comfortable with it now. Two things to spot:

The entire upper-right triangle is blank — that's the causal mask drawn out: a token can't attend to the future.
The row for "it" lights up brightest on "animal" — the pronoun resolution we traced by hand, now visible directly in the weights.

A 7x7 attention heatmap for the sentence 'The animal ran because it was tired.' Rows are query tokens (the token attending), columns are key tokens (attended to); darker cells mean more attention. The lower triangle is filled with weights while the upper-right triangle is blank, illustrating the causal mask. The cell in the 'it' row and 'animal' column is highlighted as the strongest off-diagonal attention.

The Whole Thing in 3 Lines

Everything we just described is, astonishingly, three lines of code. For the symbol-inclined, here is single-head self-attention in full:

import numpy as np

def softmax(x):
    e = np.exp(x - x.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

def self_attention(Q, K, V):
    scores  = Q @ K.T / np.sqrt(K.shape[-1])   # ① relevance: each query · every key
    weights = softmax(scores)                  # ② attention weights (each row sums to 1)
    return    weights @ V                       # ③ weighted blend of the values

# Q, K, V are just the token vectors passed through three learned matrices.
# The famous formula is literally this:  softmax(Q·Kᵀ / √d) · V

Read it back against the intuition: Q @ K.T is "compare every query to every key" (relevance), softmax turns those into weights, and @ V is the weighted blend. The √d just keeps the numbers from getting too large (stability). That famous one-liner — softmax(QKᵀ/√d)V — is the entire heart of the Transformer. You now understand it.

Why This Is So Powerful

Two properties (from last lesson) come directly from this design:

Long-range, direct connections. "it" can attend straight to "animal" no matter how many words apart they are — there's a direct path between any token and any earlier one. Old RNNs had to pass information word-by-word, losing it over distance.
Fully parallel. Every token computes its attention at the same time (it's all matrix multiplies), so it trains efficiently on modern hardware — the key to scaling.

And crucially, attention is content-based: the weights depend on what the tokens mean, learned from data — not on fixed positions or hand-written rules. That flexibility is why it generalizes so well.

🧪 Try It Yourself: Watch Attention

Self-attention is easiest to see. Click any query word below and watch which other words it pays attention to. Click “it” — notice it attends mostly to “animal” (that's the model resolving what it refers to). Click “tired” — it looks back at “it”/“animal” too. This is the model wiring up relationships across the sentence, all at once. (Notice the grid's upper-right stays blank — that's the causal mask you just learned.)

Attention heatmap — click a word, see what it attends to.

Mental-Model Corrections

It's a soft blend, not a hard pick. "it" doesn't select "animal"; it takes a weighted mixture (mostly animal). Many tokens contribute a little.
Q, K, V are learned, not designed. They're the token vector times three trained matrices. The model learns what makes a good query/key/value for the task.
One attention sees one kind of relationship. A single attention can track, say, "what does this pronoun refer to." Real models run many attentions in parallel to capture different relationships at once — that's multi-head attention, the very next lesson.
"Attention" isn't human attention. It's weighted averaging guided by learned relevance — a powerful metaphor, but it's linear algebra, not awareness.

Key Takeaways

Self-attention rebuilds each token's vector as a weighted blend of the other tokens — paying more attention to the relevant ones.
It decides the weights via Query · Key · Value, a soft database lookup: a token's Query matches every token's Key → softmax weights → blend the Values.
The famous formula softmax(Q·Kᵀ/√d)·V is just compare → weight → blend.
In LLMs attention is causal: a token attends only to itself and earlier tokens.
It gives direct long-range connections and full parallelism — the roots of scaling.

Next: one attention captures one kind of relationship. Real models run several at once — multi-head attention — and we'll also plug a gap you may have noticed: how does the model even know the order of the words? (Positional encoding.)