Attention Is All You Need

Picture a relay race where each runner can start only when the one before hands over the baton. Adding runners doesn't make the race any faster. That is how a recurrent neural network reads a sentence: to process word 50 it must first finish words 1 to 49, because each step needs the summary produced by the step before.

Two problems follow. A GPU, built to do thousands of things at once, sits mostly idle while it waits. And what the first word said reaches the last word only after surviving every hand-off in between, so long-range connections fade.

Others had tried to escape the relay. Convolutional models read words through small overlapping windows, which run in parallel, but two distant words only meet after many stacked layers. The Transformer's bet was more radical: drop the step-by-step memory entirely, and let every word connect to every other word directly.

“This inherently sequential nature precludes parallelization within training examples.”Vaswani et al. (2017), §1

You walk into a library and type a search: your query. Every book has a catalogue card describing it, its key, and the book itself on the shelf, its value. The librarian compares your search with every card and hands you not one book but a blend: mostly the best match, a little of the next best.

In a Transformer, every word plays all three parts at once. Each word searches, is catalogued and gets borrowed. This is self-attention, and it's the only place in the whole model where words exchange information.

“An attention function can be described as mapping a query and a set of key-value pairs to an output.”Vaswani et al. (2017), §3.2

Let's do the whole calculation for the word it, with numbers small enough to check in your head. Each word gets three short lists of numbers (vectors): a query, a key and a value. Real models learn these, and use 64 numbers each. We'll use four, and two for the values so we can draw them.

import math
q = [1, 1, 1, 1]                                   # the query for "it"
keys = {"animal": [1, 1, 1, 1], "tired": [1, 1, 0, 0], "street": [1, 0, 0, 0]}
values = {"animal": [1, 0], "tired": [0, 1], "street": [1, 1]}

scores = {w: sum(a * b for a, b in zip(q, k)) / math.sqrt(len(q)) for w, k in keys.items()}
total = sum(math.exp(s) for s in scores.values())
shares = {w: math.exp(s) / total for w, s in scores.items()}
# {'animal': 0.63, 'tired': 0.23, 'street': 0.14}

out = [sum(shares[w] * values[w][i] for w in values) for i in range(2)]
# [0.77, 0.37]

One attention can only ask one kind of question at a time. So give the same sentence to eight readers, each with a different coloured highlighter. One marks who did what, another which pronoun points where, another the word just before. Afterwards, staple their notes together. That's multi-head attention: eight small attentions running side by side, each free to learn its own question.

The paper's Table 3 tested the trade-off. With the total width fixed, a single head scored 0.9 BLEU worse than eight, and 32 narrow heads also did worse: more heads help, but only up to a point. Chapter 10 charts it.

“Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions.”Vaswani et al. (2017), §3.2.2

The translation model has two halves. The encoder reads the whole English sentence and writes rich notes about every word. The decoder writes the German sentence one word at a time. The same look-around-and-blend tool is used in three places, differing only in who asks and who answers:

During training the model sees the whole correct German sentence at once, which is what makes training parallel. To stop the decoder cheating by reading the word it's meant to predict, the future is blanked out with a causal mask.

Attention has a blind spot: it has no sense of order. “Dog bites man” and “man bites dog” contain the same words, so without help they'd get the same result. The fix is to stamp each word with its position before anything else happens.

The paper's stamp is a wall of clocks whose hands turn at different speeds: a hand that spins every few words, one that takes dozens, one that takes thousands. Read all the hands together and you know exactly where you are. Better still, moving three words along always turns each hand by the same amount, which makes “three words later” easy for the model to recognise.

The paper also tried learned position vectors and got nearly identical results (Table 3, row E: 25.7 BLEU against 25.8). Most modern models use RoPE instead, which rotates queries and keys by an angle set by position: the same clock-hand idea, applied inside attention.

Now assemble the parts. Words become vectors, positions are stamped on, and then the same layer repeats six times: attention, where words talk to each other, then a small feed-forward network, where each word thinks privately about what it heard. Around every step runs a shortcut, the residual connection: each layer writes corrections in the margin rather than rewriting the page. Then layer normalization tidies the numbers so nothing grows out of control.

How many phone calls does it take for news to travel between two people? Along a chain of friends, as many calls as there are people between them. On a group call, one. The paper compares layer types on exactly this, plus how much work each does.

A new driver pulls out of the driveway slowly, speeds up on the open road, then eases off near the destination. The learning rate follows the same shape. It starts tiny because an untrained model's gradients point in wild directions, climbs during a warmup, then decays so the model can settle into fine detail. The optimizer is Adam.

Two more tricks fight overfitting. Dropout (P = 0.1) zeroes a random tenth of each sub-layer's output during training. Label smoothing (ε = 0.1) marks the right answer as 90% right rather than 100%. The paper notes that label smoothing makes perplexity worse, because the model learns to be less sure, but improves BLEU: a training metric and a quality metric disagreeing.

BLEU scores a translation by how many of its word sequences also appear in a professional translation, on a 0 to 100 scale. A gain of one or two points counts as a clear step forward. The Transformer didn't just win; it won while costing less to train.

The paper closes by planning to take attention to images, audio and video, to study attention restricted to local windows for long inputs, and to make generation less sequential. All three happened. The core (attention and feed-forward layers wrapped in residual connections) is unchanged to this day. Nearly every detail around it has been tuned.

“In this work, we presented the Transformer, the first sequence transduction model based entirely on attention.”Vaswani et al. (2017), §7

Each one checks an idea, not a fact you could look up. Pick an answer to see why it's right or wrong.