Subscribe
AI
6 min read

What Is Attention? How Language Models Decide What to Read

Every explainer of a large language model eventually reaches the same sentence: the model works by attention. It is the mechanism at the center of the transformer architecture, the reason the 2017 paper that introduced it was titled "Attention Is All You Need," and the thing that makes a model feel like it understands which words in a sentence relate to which other words. Here is what attention actually is, where it came from, and what it costs. The problem attention solves Before 2014, the d

What Is Attention? How Language Models Decide What to Read

Every explainer of a large language model eventually reaches the same sentence: the model works by attention. It is the mechanism at the center of the transformer architecture, the reason the 2017 paper that introduced it was titled "Attention Is All You Need," and the thing that makes a model feel like it understands which words in a sentence relate to which other words. Here is what attention actually is, where it came from, and what it costs.

The problem attention solves

Before 2014, the dominant way to process language with neural networks was recurrence: read the sentence one word at a time, keep a running hidden state, pass it along. The state is a compression. By the time the network reaches word 30, everything about words 1 through 29 must survive inside a single fixed-size vector.

Dzmitry Bahdanau and colleagues at Universite de Montreal identified this as the bottleneck for neural machine translation. Their 2014 paper Neural Machine Translation by Jointly Learning to Align and Translate proposed a fix they called attention: instead of forcing the decoder to work only from the compressed summary, let it "automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word." The decoder could spread its gaze across all the encoder's positions at each step, weighting some more than others. The method also made the weights inspectable, and the resulting alignment maps matched human intuition about which words translate to which.

Three years later, a team of eight Google Brain and Google Research engineers rebuilt machine translation around this one idea and threw away the recurrence entirely. The resulting architecture, the transformer, was published in June 2017 as a 100-million-parameter model that beat the then-best translation systems while training in a fraction of the time (Wikipedia's account keeps the timeline). Every major language model since, from GPT onward, is built from stacks of the same attention layers.

Queries, keys, and values

Attention is best understood as a soft lookup table, an analogy Jay Alammar popularized in The Illustrated Transformer. In a conventional lookup, a query matches a key exactly and you receive the corresponding value. Attention relaxes all three steps: the query and key are vectors, the match is a similarity score rather than a yes-or-no, and instead of returning one value the operation returns a weighted average of every value in the sequence.

A single query block sending weighted arcs to a row of key and value bars.

Concretely, the paper describes the mechanism as follows. Each token's current representation is multiplied by three learned matrices to produce a query vector, a key vector, and a value vector. The token's query is compared by dot product against every token's key. Those raw scores are divided by the square root of the key dimension, then passed through a softmax, which turns them into a probability distribution: nonnegative weights that sum to 1. The output is the value vectors averaged with those weights.

Two details in that recipe are load-bearing. The division by the square root of the key dimension, which the authors name scaled dot-product attention, keeps large dot products from pushing the softmax into saturated regions where gradients vanish. And the softmax means the token never fully ignores anything: early in training, weights are diffuse, and the operation mostly averages. Sharpening the weights into selective focus is what learning does.

Take the sentence from Alammar's walkthrough: "The animal didn't cross the street because it was too tired." When the model processes the word "it," attention learns to place a large weight on "animal" and small weights on "street" and everything else. No rule says to do this; the weights come from training. One attention head computing this kind of referring-expression link is a single weighted average over other tokens' values, and it happens for every token in parallel.

Why one lookup per token is not enough

Averaging is the weakness of a single attention operation: if a token needs to track several distinct relationships at once, say what "it" refers to and which verb the subject belongs to, one probability distribution has to blend them into a mush of weights. The transformer paper's answer is multi-head attention. Instead of performing the attention function once, the model performs it h times in parallel on different learned linear projections of the queries, keys, and values into smaller subspaces, then concatenates the outputs. The base model in the paper used 8 heads over a 512-dimensional stream, so each head saw a 64-dimensional slice and the heads could specialize. The authors put it plainly: multi-head attention "allows the model to jointly attend to information from different representation subspaces," which averaging with a single head inhibits.

In practice, trained models do develop something like specializations, and interpretability research has spent years cataloguing which heads do what. That is a story for another post; the point here is the design logic: many narrow, cheap attentions beat one broad, fat one.

Why attention replaced recurrence

The shift to attention changed two things that matter for scaling.

The first is parallelism. A recurrent network must process word 1 before word 2 because the hidden state flows forward sequentially, which makes training impossible to parallelize across time. Self-attention computes all of a layer's outputs at once: every token's query is dotted against every token's key in a single matrix multiply. The paper's own complexity table contrasts this with recurrence, where sequential operations scale with sequence length versus attention's constant number. Constant sequential operations plus massive matrix multiplies is exactly what GPUs and TPUs are built for, and training data and compute could finally scale.

The second is path length. In a recurrent model, information from the beginning of a sentence reaches the end through one transformation per step, and gradients have to survive the same gauntlet. In a self-attention layer, any position can send information to any other position in a single step, which the authors credit with making "long-range dependencies easier to learn."

There is a catch, and it appears in the same table. Attention compares every token against every token, so the computation and the score matrix grow with the square of the sequence length. Double the context and one layer's attention work quadruples. This is the origin of the quadratic bottleneck that the entire applied-research subfield surveyed in our KV cache explainer and the many FlashAttention variants exists to fight.

Generation, masks, and the keys you cache

Language models today are decoders: they generate one token at a time, and each new token must attend to the whole text so far. Two constraints shape how attention runs at inference time.

The first is causality. A token may look at earlier tokens but never at later ones, since those do not exist yet. The transformer paper implements this by masking the disallowed scores to negative infinity before the softmax, which drives their weights to exactly zero. Decoder-only models apply the same mask to every layer.

The second is the cache. When the model generates token 101, it needs the keys and values of tokens 1 through 100, which it already computed at previous steps. Rather than recompute them every step, serving stacks keep them in memory as the key-value cache (the KV cache). That is why inference memory grows linearly with context length and batch size, and it is the reason long contexts cost real money per request even before the quadratic term does. Hugging Face's cache strategies documentation describes this trade of memory for speed in operational terms.

What attention is not

Two caveats keep the concept honest.

Attention weights are not explanations. A head that puts 90% of its weight on one token is not asserting that token is the reason for the output; the value it contributes is then mixed with feed-forward processing at every layer. Researchers who want causal claims intervene on activations instead of reading weights. This distinction is the whole subject of the activation patching literature.

And attention is content-dependent routing, nothing more. It decides what information reaches each position at each layer. The model's stored knowledge lives mostly in the feed-forward layers, and attention's job is wiring, not memory. "Attention Is All You Need" was architecture marketing; a transformer without its feed-forward blocks is not a language model.

The short version

Attention is a learned, differentiable lookup: each token builds a query, compares it against every other token's key, and returns a weighted average of values. It replaced recurrence because it parallelizes and shortens information paths, and it dominates transformers because relationships between words are exactly the problem it was invented for. The price is quadratic in sequence length and a growing key cache at generation time, which is why almost every performance innovation in the field, from grouped-query attention to speculative decoding, is some clever renegotiation of that price.

Sources

AILLMeducational