Transformers Explained: Attention Is All You Need in Plain English

In June 2017, eight researchers from Google Brain, Google Research and the University of Toronto posted a paper with a short title: Attention Is All You Need. It introduced the Transformer, a way to build a neural network for language that dropped the step-by-step reading that earlier models relied on. Later models such as BERT, GPT-3 and the Vision Transformer, covered below, built on it.

This explainer walks through what the paper proposed, how attention works in plain English, what the original results were, and why the design spread far beyond translation. No math background is needed. Where we use a number, it comes from the paper itself.

The problem the paper set out to solve

Before the Transformer, the strongest models for tasks like translation were built mostly from recurrent neural networks. The paper describes these models as "complex recurrent or convolutional neural networks in an encoder-decoder configuration."

A recurrent network reads a sentence one word at a time, and each step depends on the step before it. That has two costs:

• It is slow to train. Because step 5 needs the result of step 4, you cannot do all the steps at once. The paper calls this "the fundamental constraint of sequential computation."
• Long-distance links are hard to learn. If a word at the start of a sentence matters for a word at the end, the signal has to pass through every step in between.

Researchers had already added a trick called attention to these models, which let the decoder look back at any part of the input. But the paper notes that, in almost all cases, attention was still used together with a recurrent network. The authors asked a simple question: what if attention did all the work, and the recurrence was removed entirely?

Their answer, in the paper's words, is "a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output."

Attention in plain English

Think of attention as a smart lookup. For every word (strictly, every token, a piece of a word), the model asks: which other words in this sentence should I pay attention to, and how much?

The paper frames this with three roles:

• Query: what the current word is looking for.
• Key: a label each word offers, describing what it contains.
• Value: the actual information each word hands over if it is chosen.

The model compares a word's query with every key. Strong matches get high scores. The scores are turned into weights that add up to one (that is the "softmax" step), and the word's new representation is a weighted blend of the values. In the paper's words, the output is a weighted sum of the values, "where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key."

A simple example: in "The cat sat on the mat because it was tired," a good attention pattern lets "it" put most of its weight on "cat." The model is not told this rule. It learns which links help during training.

Why "scaled" dot-product attention

The paper calls its version "Scaled Dot-Product Attention." The comparison between a query and a key is a dot product, a quick multiply-and-add. The authors divide each score by the square root of the key size before the softmax. Their reason: with large vectors, raw dot products "grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients," which makes learning stall. Scaling keeps the numbers in a useful range.

They also chose dot products over an older method called additive attention because dot-product attention "is much faster and more space-efficient in practice, since it can be implemented using highly optimized matrix multiplication code."

Multi-head attention

One attention pattern can only blend information one way. The paper runs several attention operations side by side, called heads. The original model uses 8 heads. Each head gets its own learned view of the queries, keys and values, so different heads can learn different kinds of links between words. The outputs are joined and mixed back together. Our multi-head attention explainer goes deeper on how the heads split the work.

The authors explain the benefit this way: "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this." Because each of the 8 heads works on a smaller slice (64 dimensions each, out of 512), the total cost is "similar to that of single-head attention with full dimensionality."

The full Transformer, piece by piece

The original Transformer was built for translation, so it has two halves.

The encoder reads the source sentence (say, English). It is a stack of 6 identical layers. Each layer has two parts: a multi-head self-attention step, where every word looks at every other word in the input, and a small feed-forward network applied to each position on its own. "Self-attention" just means the queries, keys and values all come from the same sentence.

The decoder writes the output sentence (say, German) one token at a time. It is also a stack of 6 layers. It has the same two parts plus a third: attention over the encoder's output, so each output word can look at the whole source sentence.

The decoder has one important rule. When it predicts the next word, it must not peek at words it has not written yet. The paper enforces this with a mask that blocks attention to later positions, so the model stays "auto-regressive": each new token depends only on the tokens before it. This left-to-right setup is the same basic idea behind modern text generators, which produce a reply one token at a time.

A few supporting parts hold the stack together:

• Residual connections and layer normalization. Around each sub-layer, the input is added back to the output and then normalized.
• A feed-forward network in every layer, with an inner size of 2,048 in the base model, which processes each position after attention has mixed in context.
• Model width. Every layer passes around vectors of size 512 in the base model.

How a model with no order knows word order

There is a catch. Attention by itself treats a sentence like a bag of words: it has no built-in sense of which word came first. The paper says it plainly: "Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens."

The fix is positional encoding. Before the first layer, each token's embedding gets a position signal added to it. The authors used sine and cosine waves of different frequencies, so each position gets a unique pattern. They also tried learned position embeddings and found the two "produced nearly identical results." They kept the sine version because it might let the model handle sequences longer than those seen in training. Later work proposed other ways to encode position, which our relative positional encodings explainer covers.

Why removing recurrence mattered

The paper compares self-attention with recurrent layers on three measures: computation per layer, how much work can run in parallel, and how far a signal must travel between two distant words.

• A self-attention layer connects all positions with a constant number of sequential steps. A recurrent layer needs a number of steps that grows with the sentence length.
• The path between any two words is one step in self-attention, but grows with distance in a recurrent network. The paper notes that shorter paths make long-range dependencies easier to learn.

The trade-off is cost per layer. Every token attends to every other token, so the work grows with the square of the sequence length. The paper lists this as O(n² · d) per layer. The paper notes that self-attention is still faster than a recurrent layer when the sequence is shorter than the vector size, which it says is most often the case for the sentence representations used in translation. For very long documents it became a real limit, and later work focused on it. The FlashAttention paper, for example, opens by noting that "the time and memory complexity of self-attention are quadratic in sequence length," and proposes a faster, memory-aware way to compute exact attention on GPUs. Our FlashAttention explainer covers the method in more detail.

What the original results showed

The paper tested the Transformer on two standard translation benchmarks from WMT 2014 and reported results with BLEU, a common score that measures overlap with human translations.

• English to German: 28.4 BLEU, which the authors say beat the existing best results, including ensembles, "by over 2 BLEU."
• English to French: 41.8 BLEU, a new single-model state of the art at the time, after "training for 3.5 days on eight GPUs."

Training cost was a big part of the pitch. The team trained on one machine with 8 NVIDIA P100 GPUs. The base model trained for 100,000 steps, about 12 hours. The larger "big" model trained for 300,000 steps, about 3.5 days. The abstract describes this as "a small fraction of the training costs of the best models from the literature."

The authors also tried the model on a different task, English constituency parsing (breaking a sentence into its grammatical structure), and report that it "generalizes well" there too, with both large and limited training data.

How one translation paper spread everywhere

The paper was about translation, but the building block was general. A few well-known follow-ups show how far it traveled:

• BERT (2018). BERT stands for "Bidirectional Encoder Representations from Transformers." It kept the encoder side and pre-trained it on unlabeled text, then fine-tuned it for tasks like question answering. The BERT paper reports new state-of-the-art results on eleven language tasks.
• GPT-3 (2020). OpenAI's GPT-3 paper describes "an autoregressive language model with 175 billion parameters" and shows that scaling up improved few-shot performance, meaning the model could do new tasks from a handful of examples in the prompt. Autoregressive generation is the same one-token-at-a-time idea as the original decoder.
• Vision Transformer (2020). Researchers showed in An Image is Worth 16x16 Words that "a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks." The same paper opens by calling the Transformer "the de-facto standard for natural language processing tasks."

The common thread is that attention does not care what the tokens are. Words, image patches and other kinds of data can all be cut into tokens, and the same machinery learns which tokens matter to which.

Key terms, in one place

• Token: a small chunk of text (often part of a word) that the model reads and writes. The paper used byte-pair encoding for English-German, with a shared vocabulary of about 37,000 tokens.
• Attention: a learned, weighted lookup where each token blends in information from other tokens.
• Self-attention: attention where the queries, keys and values all come from the same sequence.
• Head: one attention operation. Multi-head attention runs several in parallel.
• Encoder / decoder: the reading half and the writing half of the original model.
• Positional encoding: a signal added to each token so the model knows word order.
• Auto-regressive: generating one token at a time, each based only on earlier tokens.

How to read the paper yourself

The paper is about 15 pages and is free on arXiv. If you want to try it:

1. Read the abstract and the introduction for the motivation.
2. Look at Figure 1 (the full model) and Figure 2 (attention and multi-head attention) before the equations. The pictures carry most of the idea.
3. Read section 3.2 on attention slowly. Equation 1 is the core of the whole design.
4. Skim section 4, "Why Self-Attention," for the comparison with recurrent layers.
5. Check Table 2 for the translation results and training cost.

You do not need to follow every symbol. If you can explain queries, keys, values and why the decoder is masked, you understand the main idea of the paper.

Sources

• Attention Is All You Need (Vaswani et al., arXiv 1706.03762)
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv 1810.04805)
• Language Models are Few-Shot Learners (GPT-3, arXiv 2005.14165)
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (arXiv 2010.11929)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (arXiv 2205.14135)