Why Transformers Won (vs RNNs/LSTMs)

We've built the Transformer piece by piece. Now the payoff question: why did it so completely replace the previous champion? For years, the best language models were RNNs and LSTMs. Then the 2017 paper "Attention Is All You Need" threw out recurrence entirely — and within a few years, everything was Transformers.

Understanding why isn't trivia. It explains the deep reasons LLMs could get so big and so good — and reveals the one tax Transformers pay (which shows up directly in your context limits and your bill).

You'll learn:

• How the old champion (RNNs/LSTMs) worked — and its two fatal flaws
• The two properties that let Transformers win: parallelism & direct attention
• The tradeoff Transformers pay (quadratic attention)
• Why all of this matters for you as an AI engineer

An RNN (recurrent neural network) reads text the way you might think you do: one token at a time, left to right, carrying a hidden state — a single running summary of everything seen so far. Read a word, update the summary, pass it to the next step, repeat.

LSTMs (and GRUs) are souped-up RNNs with internal gates that decide what to remember and what to forget, so they hold information for longer. For years, LSTMs were the state of the art in translation, speech, and text.

The whole design rests on one thing: recurrence — each step depends on the previous step's output. That's also exactly where it falls apart.

Because each step needs the previous step's hidden state, an RNN must process token 1, then token 2, then token 3… You cannot parallelize across the sequence — step 5 literally can't start until step 4 finishes.

That's a death sentence in the age of GPUs. Modern hardware is built to do thousands of operations at once, but recurrence forces them into a single-file line. Training on internet-scale data becomes impossibly slow. And scale is everything in modern AI — if you can't train fast on huge data, you can't build a frontier model. RNNs simply couldn't keep up.

The second flaw is about memory. Everything an RNN knows about the past is squeezed into one fixed-size hidden state — an information bottleneck. The whole first half of a paragraph has to be compressed into the same little vector that's also tracking the current word.

The result: long-range dependencies fade. By the time the model reaches the end of a long passage, the beginning has been overwritten and diluted (the classic vanishing-gradient problem). If a pronoun refers to a noun 80 words earlier, an RNN struggles to connect them, because that information had to survive 80 sequential hand-offs. LSTM gates helped, but didn't truly solve it.

The Transformer's design (which you now understand in full) fixes both flaws at once:

• Parallelism. With no recurrence, every token is processed at the same time — attention is just big matrix multiplications. That maps perfectly onto GPUs, so you can train on enormous data efficiently. This is the property that unlocked scale.
• Direct long-range connections. Via self-attention, any token can attend directly to any earlier token — one "hop," no matter how far apart. "it" reaches straight back to "animal" 80 words ago with no decay. There's no single bottleneck state; every token has full access to all the others.

So the two things RNNs were worst at — speed/scale and long-range memory — are exactly the two things attention is best at. That's why it wasn't a close fight.

Transformers aren't free lunch — they pay a real tax, and you should know it.

Because every token attends to every other token, the attention computation grows with the square of the sequence length: n tokens → roughly n² comparisons. Double your context, and attention costs 4×; 10× the context, 100× the attention work. This O(n²) scaling is the deep reason that context windows are limited and long prompts get expensive (recall the token-cost lesson — now you know why it's not just linear).

RNNs, by contrast, were O(n) — linear in length — but sequential. So the trade the field made was: accept quadratic cost in exchange for parallelism and global context. It was overwhelmingly worth it — but it's why "make the context longer" is a hard, active research problem (efficient-attention variants, and even RNN-flavored comebacks like state-space models such as Mamba, are all about beating this n²).

This isn't just history — the Transformer's DNA shows up in your daily work as an AI engineer:

• Context limits exist because attention is O(n²). That's why models have a max context window, and why stuffing everything into the prompt eventually breaks down.
• Long inputs cost more than linearly — a reason RAG (retrieve only what's relevant) often beats dumping huge documents into the prompt.
• Scale worked because of parallelism — the entire "just make it bigger" era of progress rode on the Transformer's trainability.

Understanding the architecture means these constraints aren't mysterious rules — they're the predictable consequences of how the machine is built.

Predict the winner. You feed a 1,000-word document to (a) an RNN/LSTM and (b) a transformer. Which processes it faster on a GPU, and why?

→ The transformer, because it processes all tokens in parallel (attention sees the whole sequence at once), while an RNN must walk token-by-token, sequentially. That parallelism is the single biggest reason transformers won — it's what made training on internet-scale data feasible.

• Transformers aren't strictly better at everything. They pay O(n²). For extremely long sequences, that cost is brutal — which is why efficient variants and state-space models (Mamba) are active research. Attention won decisively for LLMs, but the story isn't fully over.
• "RNNs are dead" is too strong. They're out of favor for frontier LLMs, but the ideas (sequential state, gating) live on, and hybrids exist.
• Attention's strength is its cost. "Look at everything, directly" is exactly what makes it powerful and exactly what makes it quadratic. You can't have the global view for free.
• Bigger context ≠ free. More context window is genuinely more expensive and slower — budget it like the finite resource it is.

• RNNs/LSTMs read sequentially with a single fixed hidden state — so they were slow (no parallelism) and forgot long-range context (information bottleneck / vanishing gradients).
• Transformers won on exactly those two axes: parallel processing (→ trainable at massive scale) and direct attention between any two tokens (→ no long-range decay).
• The tradeoff is O(n²) attention — every token attends to every other — which is why context windows are limited and long prompts are expensive.
• These properties explain real constraints you'll hit: context limits, super-linear cost, and why scale worked.

Next: theory complete — time to build. In the final lesson of this section you'll write a tiny GPT from scratch, turning every concept here into ~100 lines of working code.