Every frontier language model you have used, GPT, Claude, Gemini, Llama, DeepSeek, is a transformer. The word does not describe a company, a product or a training method. It describes the circuit layout, the arrangement of matrix multiplications that turns a sequence of tokens into a prediction of the next one. Two models from different labs can share this layout while behaving completely differently, because the layout says how computation is organized, not what the model knows.
The design comes from a single paper: "Attention Is All You Need," submitted to arXiv in June 2017 by eight researchers from Google Brain, Google Research and the University of Toronto (arXiv:1706.03762). The original model was built for machine translation and had 65 million parameters. Its direct descendants now run to hundreds of billions.
The problem it solved
Before 2017, sequence models were recurrent. An RNN reads a sentence one token at a time and passes a hidden state from step to step. The paper's own framing of the weakness is blunt: recurrent models "cannot be parallelized due to their sequential nature" (Vaswani et al., 2017). To compute the state after the 500th token, you must first compute all 499 before it. More processors do not help, because the computation itself is a queue.
Sequential processing also makes long-range influence fragile. Information connecting the first and last words of a paragraph must survive hundreds of state updates, and in practice the signal decays.
The transformer removes recurrence entirely. Every token in the sequence is processed at the same time, and any token can reach any other token in a single computation step. The authors showed the effect on training time: their big model finished 3.5 days on eight NVIDIA P100 GPUs and beat every published translation result at the time, including ensembles (Vaswani et al., 2017).

The core mechanism: attention
Attention answers one question for each token: which other tokens in the sequence should I look at, and how much? It does this with three vectors per token, called query, key and value, each produced by a learned linear projection of the token's current representation (Vaswani et al., 2017).
The mechanics are simple enough to state in two sentences. A token's query is compared by dot product against every token's key, the scores are normalized with softmax, and the token's new representation is the weighted average of all values. When you read "bank" in "she deposited cash at the bank," the right question is which earlier words the model should pull toward it; attention computes that weighting for all pairs of words in one batched matrix multiply.
The original model runs this operation in parallel with eight heads, each projecting into a smaller 64-dimensional space (Vaswani et al., 2017). Different heads end up sensitive to different patterns; later interpretability work found heads tracking syntax, nearby words and other regularities (a broad discussion is in Wikipedia's transformer article)). Attention is the mechanism that lets a transformer connect a pronoun to its antecedent 2,000 tokens away without any recurrence in between.
What a transformer block contains
One layer of a modern decoder stacks four pieces:
- Token embeddings plus positional information. Text is cut into subword tokens (byte-pair encoding is the standard scheme, Sennrich et al., 2015), each mapped to a vector. Since attention itself ignores order, positions are injected separately, via sinusoidal encodings in 2017 (Vaswani et al., 2017) and increasingly via rotary embeddings today (Su et al., 2021).
- Self-attention. The mixing step described above. Within a decoder, attention is masked so each token can look backward at earlier tokens only, which is what makes autoregressive generation well defined (Vaswani et al., 2017).
- A feed-forward network. Two linear transformations with a nonlinearity between them, applied independently at each position (Vaswani et al., 2017). Most of a modern LLM's parameters live in these layers.
- Residual connections and normalization. Each sub-layer's output is added back to its input and normalized (Vaswani et al., 2017). The residual is why depth works: information and gradients have a direct highway through the stack, so later layers modify the representation rather than rebuild it.

The 2017 model used six of these blocks in the encoder and six in the decoder. Modern LLMs keep only the decoder half and stack tens to over a hundred blocks.
Why the design scaled
The transformer won because its two expensive operations, matrix multiplies over all token pairs and matrix multiplies over feature dimensions, map almost perfectly onto GPUs and TPUs. Nothing waits for anything else within a training step. That hardware fit, more than any cleverness in attention, is what made billion-parameter pre-training practical.
The scale change since 2017 is easy to trace in the papers themselves: 65 million parameters in the original model (Vaswani et al., 2017), 175 billion in GPT-3 (Brown et al., 2020), 405 billion in Llama 3 (Meta, 2024). Surveys of the field note that the field converged on the decoder-only variant of the architecture as the dominant template for large language models (Zhao et al., 2023).
Scaling was not free, and most post-2017 research is about paying down two specific bills of the original design.
Bill one: attention grows quadratically. Computing all token pairs costs time and memory proportional to sequence length squared (Vaswani et al., 2017; see also the survey of efficient transformers). FlashAttention computes the same exact attention with IO-aware tiling so the quadratic score matrix never fully lands in slow memory (Dao et al., 2022).
Bill two: generation re-reads the past. A decoder caches the keys and values it computed so it does not recompute them each step, but that KV cache grows with sequence length and is served from memory bandwidth, which the paper on GQA identifies as the severe bottleneck of autoregressive inference (Ainslie et al., 2023; for the caching scheme itself, Pope et al., 2022). The standard fixes shrink the cache: grouped-query attention shares key-value heads across groups of query heads (Ainslie et al., 2023), and DeepSeek-V2's multi-head latent attention compresses them into a low-rank form (DeepSeek, 2024). Llama 3 70B uses 64 query heads with 8 KV groups, cutting cache size eightfold (Meta, 2024).
For the feed-forward layers, the change is substitution: mixture-of-experts models swap the dense block for many expert blocks plus a router that activates a few per token, decoupling parameter count from per-token compute (Fedus et al., 2021).
Three things people get wrong
A transformer is not a model. It is a template. Two models with identical transformer stacks differ by their weights, which come from their training data and objective.
Attention is not memory in the everyday sense. It reads the current context window each forward pass. When the window closes, nothing persists; parametric knowledge lives in the weights, not in attention.
Order is not native. Attention treats its inputs as an unordered bag of vectors, which is exactly why positional encodings are required, and why changing token positions can change model behavior in ways that surprised early users of long-context models (Vaswani et al., 2017; Liu et al., 2023 on lost-in-the-middle position effects).
The 2017 paper solved a translation benchmark. What it actually shipped was the standard substrate of the next decade of language models, and the open research questions about it, context length, serving cost, position handling, are still the same two bills, paid in new ways.
Sources
- Vaswani et al., "Attention Is All You Need," NeurIPS 2017, arXiv:1706.03762
- Sennrich et al., "Neural Machine Translation of Rare Words with Subword Units," arXiv:1508.07994
- Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding," arXiv:2104.09864
- Brown et al., "Language Models are Few-Shot Learners" (GPT-3), arXiv:2005.14165
- Meta, "The Llama 3 Herd of Models," arXiv:2407.21783
- Zhao et al., "A Survey of Large Language Models," arXiv:2303.18223
- Dao et al., "FlashAttention," arXiv:2205.14135
- Tay et al. (via efficient-transformers coverage), "Efficient Transformers: A Survey," arXiv:2107.06799
- Pope et al., "Efficiently Scaling Transformer Inference," arXiv:2211.05102
- Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints," arXiv:2305.13245
- DeepSeek-AI, "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model," arXiv:2405.04434
- Fedus et al., "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity," arXiv:2101.03961
- Liu et al., "Lost in the Middle: How Language Models Use Long Contexts," arXiv:2307.03172
- Wikipedia, "Transformer (deep learning)," en.wikipedia.org/wiki/Transformer_(deep_learning))
