Transformers and Attention

Every model people talk about today — GPT, Llama, Qwen, Claude — rests on the same architecture, introduced in 2017 in a paper with a blunt title: Attention Is All You Need. Understanding what changed back then explains why these models work.

The video is in Italian; this page is the written version in English.

The problem before

Until 2017, language models used recurrent networks, RNNs. They read text one word at a time, in sequence, the way we do.

"The cat my friend gave me yesterday is very sweet"

The → cat → my → friend → gave → me → yesterday → is → very → sweet
                                                              ^
                                          here you must still remember
                                          we are talking about the cat,
                                          eight words later

Three problems follow. Long-range dependencies fade: by the time you reach “sweet”, the subject has blurred. Slowness is structural: one token at a time, in order, with no parallelism — and training data runs to billions of tokens. And during training the error signal dissolves as it travels back through the sequence, making distant relationships hard to learn.

The idea: look at everything at once

The Transformer move is simple to state: instead of reading word by word, look at the whole sentence at once and decide what matters.

Picture a table with ten people. RNNs are Chinese whispers: the message passes from mouth to mouth and warps. Attention is a conversation where anyone can address anyone else directly, immediately.

For each word the model asks: which other words do I need to attend to in order to understand this one?

"The cat sat on the rug because it was tired"

To understand "it":
  cat  → HIGH attention   (who was tired? the cat)
  rug  → LOW attention    (the rug is not tired)
  sat  → MEDIUM attention (it explains the situation)

Query, Key, Value

The mechanism uses three views of every token, and the library analogy makes them immediate:

• Query — the question a token asks: “I’m looking for books on photosynthesis”
• Key — the label each token advertises: “Biology”, “Chemistry”, “Cooking”
• Value — the actual content of the book

The score comes from Query meeting Key: how relevant that book is to that question. Then you take the Value, weighted by that score. Repeated for every word against every other word, this is the heart of self-attention.

Why “multi-head”

A single attention captures one kind of relationship at a time. Transformers run several in parallel — the heads — and each learns to look at something different: one follows subject-verb agreement, another tracks what pronouns refer to, another semantic relatedness. The results are then merged.

It is the difference between rereading a sentence with one criterion and rereading it with several at the same time.

The price

Attention compares every token with every other token. For a sequence of length n the cost grows with n²: doubling the input quadruples the work.

That is why very large context windows are expensive in memory and time, and why much of the recent research targets exactly this bottleneck.

Why the first token is slow and the rest are not

Anyone running a model locally notices two different timings: an initial wait, then a steady stream of words. The explanation lies in a structure that gets little attention: the key and value cache.

During generation, for each new token the model would have to recompute Key and Value for all preceding tokens. That would be enormously wasteful, because those values never change. So they are kept.

FIRST TOKEN
  process the entire prompt        → slow, and grows with length
  store Key and Value of each token

SUBSEQUENT TOKENS
  compute only those of the new token  → fast and constant
  reuse everything else from the cache

Three consequences you can feel. The time to the first word depends on prompt length; the time between later words does not. The cache occupies memory and grows with context, which is why a long conversation can exhaust memory even though the model alone fitted comfortably. And changing the start of the prompt invalidates everything: keep fixed parts first — system instructions, documents — and variable parts last, because several systems reuse an already-processed prefix.

For scale: on long contexts this cache reaches several gigabytes, sometimes more than the model itself.

How that memory is reduced

If the cache is the problem, the obvious move is to shrink it. In the classic version every attention head has its own Keys and Values, so memory multiplies by the number of heads.

Variant	How it works	Effect
Classic	Every head has its own Key and Value	Maximum memory, maximum quality
Multi-query	All heads share a single set	Minimum memory, some loss
Grouped (GQA)	Heads are split into groups, one set per group	Nearly classic quality, far less memory

The third is now standard: Llama, Qwen and Mistral all use it. One of those choices invisible to users and decisive in practice, because it is what makes wide context windows workable on ordinary hardware.

The quadratic cost, sidestepped

Complexity growing with the square of the length is theoretical and cannot be removed. What can be removed is the real bottleneck, which is not computation but memory traffic.

The naive implementation builds the entire attention matrix and writes it to memory. With long sequences that matrix is enormous, and moving it back and forth costs more than the arithmetic. The technique known as flash attention processes it in blocks instead, keeping pieces in the fast memory near the processor and never materialising the whole thing.

Naive:   compute the whole matrix → write to memory → read back
Blocked: work one piece at a time in fast memory → write only the result

The operation count is identical, the time drops considerably. Not a specialist detail: it is one of the reasons context windows of tens of thousands of tokens became usable, and it runs almost everywhere without users knowing.

What changed since 2017

The original architecture is still recognisable, but several internal pieces have been replaced. The names recur on model cards, so they are worth recognising.

In the original paper	Today	Why
Normalisation after the block	RMSNorm, before the block	Cheaper, more stable training
ReLU activation	SwiGLU	Better quality at equal parameter count
Absolute positional encoding	RoPE	Handles long sequences better
Dense attention	GQA	Less cache memory

None of these touch the core idea: they are eight years of optimisations layered on the same frame. That the frame held without radical replacement, in a field moving this fast, is remarkable in itself.

In short

Concept	In one line
RNN	Reads in sequence, loses distant links, cannot parallelise
Self-attention	Every token looks at all others and weighs how much they matter
Query / Key / Value	Question, label and content: the score comes from these
Multi-head	Several attentions in parallel, each on a different relationship
n² cost	The bill grows with the square of the input length
Key/value cache	Makes post-first tokens fast, but occupies memory
GQA	Grouped heads: far less memory, quality nearly intact
Flash attention	Processes attention in blocks: same maths, far less traffic