Decoder-Only vs. Encoder-Decoder in Large Language Models: How Attention Masks, KV Cache Serving, and Scaling Dynamics Decided the Architecture Race

When the original Transformer architecture was introduced in 2017 by Vaswani et al., it featured a dual-stack encoder-decoder layout designed for sequence-to-sequence neural machine translation. Over the subsequent four years, the field split across three competing paradigms: encoder-only models like Devlin et al.'s BERT for understanding, encoder-decoder models like Raffel et al.'s T5 and Lewis et al.'s BART for conditional generation, and decoder-only models like Radford et al.'s GPT series for autoregressive generation.

Today, nearly every frontier and open-weight foundation model, from LLaMA and Mistral to GPT-4, Qwen, and DeepSeek, relies exclusively on the decoder-only architecture. This convergence was not accidental. It was driven by three decisive factors: pre-training token density, key-value (KV) cache serving simplicity, and the mechanics of in-context learning.

The Three Architectural Paradigms

To understand why decoder-only architectures dominated, it is necessary to examine how attention masks and sequence processing differ across the three structural layouts.

1. Encoder-Only (BERT, RoBERTa)
Input: [Token 1] <---> [Token 2] <---> [Token 3] <---> [Token 4]
Mask: Full Bidirectional (All tokens attend to all tokens)

2. Encoder-Decoder (T5, BART, Original Transformer)
Encoder: [Token 1] <---> [Token 2] (Bidirectional)
                   |           |   (Cross-Attention)
                   v           v
Decoder: [Target 1] ---> [Target 2] (Causal Auto-regressive)

3. Decoder-Only (GPT, LLaMA, Mistral, DeepSeek)
Input + Output: [Prompt 1] ---> [Prompt 2] ---> [Token 1] ---> [Token 2]
Mask: Lower-Triangular Causal Mask (Tokens only attend to past positions)

In standard scaled dot-product attention, the attention weights are computed as:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + M\right)V$

The matrix $M$ is the attention mask that defines which tokens can exchange information with one another.

Encoder-Only Models: Set $M_{ij} = 0$ for all token positions $i$ and $j$ . Every token attends bidirectionally to every other token. While this produces rich contextual representations for classification or sequence labeling, encoder-only models cannot generate coherent autoregressive text without computationally inefficient iterative sampling.
Encoder-Decoder Models: Use two distinct sub-networks. The encoder runs bidirectional self-attention ( $M_{ij} = 0$ ). The decoder applies causal self-attention ( $M_{ij} = -\infty$ for $j > i$ ) to prevent looking ahead, combined with cross-attention layers where decoder query vectors $Q_{\text{dec}}$ attend over the final encoder key-value representations ( $K_{\text{enc}}, V_{\text{enc}}$ ).
Decoder-Only Models: Consist of a single stack of identical transformer blocks governed by a lower-triangular causal attention mask ( $M_{ij} = -\infty$ for $j > i$ ). Every token can attend only to preceding tokens and itself.

A hybrid variant known as Prefix LM (or non-causal decoder) allows bidirectional attention across the input prompt ( $1 \le i, j \le k$ ), while enforcing causal masking for all newly generated continuation tokens ( $i > k$ ).

Pre-Training Dynamics: Supervision Density and Gradient Flow

One of the primary advantages of decoder-only architectures lies in pre-training sample efficiency.

In masked language modeling (MLM) and span-corruption objectives (used by BERT, T5, and BART), the model masks a subset of tokens, typically 15%, and attempts to reconstruct them:

$\mathcal{L}_{\text{span}} = -\sum_{i \in \text{masked}} \log P(x_i \mid x_{\setminus \text{masked}})$

Under this regime, 85% of the input tokens in any given training sequence produce no direct prediction loss. While bidirectional self-attention allows the model to build contextual representations of the unmasked tokens, the gradient signal per forward-backward pass is sparse.

In contrast, autoregressive causal language modeling (CLM) computes the cross-entropy loss at every position across the entire sequence length $L$ :

$\mathcal{L}_{\text{CLM}} = -\sum_{t=1}^{L} \log P(x_t \mid x_{<t})$

Every single token acts simultaneously as a target for the preceding prefix and as context for all subsequent tokens. For a sequence of 4,096 tokens, a causal decoder computes 4,095 next-token loss gradients in a single forward-backward pass.

Large-scale empirical evaluations by Wang et al. (2022) demonstrated that when models are trained under purely unsupervised pre-training budgets, causal decoder-only models systematically outperform encoder-decoder and prefix-LM architectures on zero-shot and few-shot benchmarks.

While encoder-decoder models can achieve strong performance when pre-training is followed by extensive supervised multitask fine-tuning (as demonstrated by Tay et al. in the UL2 study), the raw scaling efficiency of next-token prediction across multi-trillion-token corpora made decoder-only models the compute-optimal choice.

Inference Serving Economics and the KV Cache Bottleneck

In production serving systems, the architectural choice dictates memory allocation, paging efficiency, and compute utilization.

Key-Value Cache Memory Flow vs. Cross-Attention Pathways

During autoregressive generation, computing attention over previously generated tokens without re-evaluating the entire sequence requires caching key and value vectors in high-bandwidth memory (HBM).

The Decoder-Only Serving Pipeline

In a decoder-only model:

Prefill Phase: The entire input prompt of length $L_{\text{prompt}}$ is processed in parallel across all layers in a single forward pass, populating the initial KV cache.
Decode Phase: Each step generates a single new token $x_t$ . The new token computes query, key, and value vectors ( $q_t, k_t, v_t$ ). The new $(k_t, v_t)$ are appended to the existing cache, and attention is computed across the accumulated history:

$\text{Attention}(q_t, K_{1:t}, V_{1:t}) = \text{softmax}\left(\frac{q_t K_{1:t}^T}{\sqrt{d_k}}\right)V_{1:t}$

This architecture provides structural uniformity:

Every transformer layer maintains an identical KV cache shape: [batch_size, num_kv_heads, seq_len, head_dim].
Memory allocators like PagedAttention in vLLM manage a single pool of non-contiguous physical memory blocks.
Prefix caching engines (such as SGLang's RadixAttention) can directly match and reuse causal KV cache prefixes across multi-turn user requests.

The Encoder-Decoder Serving Pipeline

In contrast, encoder-decoder serving introduces dual-state memory fragmentation:

The encoder processes the prompt $L_{\text{enc}}$ and writes static encoder KV caches ( $K_{\text{enc}}, V_{\text{enc}}$ ) across all layers.
The decoder maintains a separate, dynamically expanding causal self-attention KV cache ( $K_{\text{dec}}, V_{\text{dec}}$ ).
At every generation step, each decoder layer executes two attention operations: causal self-attention over $(K_{\text{dec}}, V_{\text{dec}})$ and cross-attention over $(K_{\text{enc}}, V_{\text{enc}})$ .

This dual cache structure complicates memory paging, increases kernel launch overhead, and makes dynamic prefix caching across multi-turn dialogues significantly more difficult to manage.

Compute Allocation Efficiency

Parameter utilization during generation also favors decoder-only architectures:

In an encoder-decoder model with $N$ total parameters ( $N/2$ in the encoder and $N/2$ in the decoder), the encoder parameters sit completely idle during the entire token generation phase.
In an $N$ -parameter decoder-only model, 100% of the parameter capacity is actively engaged during both prompt ingestion and every subsequent generation step.

In-Context Learning and the Unified Text Interface

The emergence of in-context learning (ICL), documented by Radford et al. in GPT-2 and Brown et al. in GPT-3, redefined how foundation models interact with users.

In-context learning frames every task as an unbroken sequence of tokens:

[System Prompt] -> [Few-Shot Example 1] -> [Few-Shot Example 2] -> [User Input] -> [Generated Answer]

Decoder-only architectures treat this entire sequence as a single autoregressive trajectory. There is no artificial architectural boundary separating "inputs" from "outputs". This uniformity enables:

Arbitrary Multi-Turn Conversations: User turns and assistant turns simply append to the running causal context without requiring re-encoding or structural re-routing.
Chain-of-Thought and Deliberation: Intermediate reasoning steps, scratchpads, and thinking tokens flow directly into the causal context, allowing subsequent reasoning tokens to condition on earlier intermediate steps.
Agentic Tool Calling: Tool execution results, environment observations, and API responses are appended sequentially to the context, maintaining full historical state within a unified KV cache.

Encoder-decoder models impose a rigid structural split. The user must decide which portion of the prompt belongs in the encoder and where the decoder generation should begin. For multi-step reasoning or agentic loops, the encoder input must either be repeatedly re-encoded or manually synchronized with decoder outputs, breaking the simplicity of the single-stream token interface.

Architectural Comparison Across Key Dimensions

Attention Mask Structure:
Encoder-Only (BERT): Fully bidirectional across all token positions.
Encoder-Decoder (T5, BART): Bidirectional self-attention in the encoder; causal self-attention and cross-attention in the decoder.
Prefix LM (UL2): Bidirectional attention within the prefix prompt; causal masking across continuation tokens.
Decoder-Only (LLaMA, GPT): Strictly lower-triangular causal attention across the entire sequence.
Pre-Training Loss and Supervision Density:
Encoder-Only: Masked language modeling (typically 15% of tokens masked). Supervision density is low (0.15 loss per token).
Encoder-Decoder: Span corruption and denoising (typically 15% of tokens corrupted). Supervision density is low (0.15 loss per token).
Prefix LM: Mixed span corruption and prefix-causal language modeling. Supervision density is moderate to high.
Decoder-Only: Autoregressive next-token prediction across all sequence positions. Supervision density is maximal (1.0 loss per token).
KV Cache and Serving Architecture:
Encoder-Only: Not applicable (non-generative).
Encoder-Decoder: Dual-state memory required (static encoder KV cache plus expanding dynamic decoder KV cache).
Prefix LM: Unified memory buffer with mixed bidirectional and causal attention kernels.
Decoder-Only: Unified single-buffer dynamic KV cache per layer, highly optimized for PagedAttention and prefix caching.
Parameter Utilization During Generation:
Encoder-Only: N/A.
Encoder-Decoder: Approximately 50% of total parameters (the encoder stack) sit idle during token decoding.
Prefix LM: 100% of model parameters participate in both prefill and decoding phases.
Decoder-Only: 100% of model parameters participate in both prefill and decoding phases.
In-Context Learning and Agentic Execution:
Encoder-Only: Unsupported for open-ended generation.
Encoder-Decoder: Rigid structural boundary requiring separation of input context and output generation.
Prefix LM: Supported across multi-task prompt formats.
Decoder-Only: Fully natural unbroken 1D token timeline, supporting multi-turn dialogue, tool calls, and chain-of-thought scratchpads.

Where Encoder-Decoder Still Holds Advantages

While decoder-only architectures dominate general language modeling, encoder-decoder models retain clear strengths in dedicated sequence-to-sequence domains:

Speech Recognition and Audio Processing: Models like OpenAI's Whisper use an audio encoder (processing mel-spectrograms bidirectionally) paired with an autoregressive text decoder. The separation allows specialized acoustic feature extraction before generating linguistic tokens.
Dedicated Machine Translation: When translating between fixed language pairs, an encoder-decoder architecture can build bidirectional source representations while producing concise target sequences.
Multimodal Encoders: Modern vision-language architectures frequently employ bidirectional vision encoders (such as ViT or SigLIP) to extract visual patches, projecting those representations directly into the input space of a causal decoder-only language model.

Summary

The triumph of decoder-only transformers was not a matter of aesthetic preference. Causal autoregression maximizes supervision density during pre-training by computing loss across every sequence token, eliminates parameter idling during inference, streamlines KV cache management in distributed serving clusters, and provides a natural, unbroken substrate for in-context learning and multi-step agent reasoning.

Sources

Vaswani, A., et al. (2017). Attention Is All You Need. arXiv:1706.03762
Radford, A., et al. (2018). Improving Language Understanding by Generative Pre-Training. OpenAI Paper
Radford, A., et al. (2019). Language Models are Unsupervised Multitask Learners. OpenAI Paper
Devlin, J., et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805
Raffel, C., et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683
Lewis, M., et al. (2020). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. arXiv:1910.13461
Wang, T., et al. (2022). What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?. arXiv:2204.05832
Tay, Y., et al. (2022). UL2: Unifying Language Learning Paradigms. arXiv:2205.05131
Brown, T., et al. (2020). Language Models are Few-Shot Learners. arXiv:2005.14165
Touvron, H., et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971