Decoder-Only vs. Encoder-Decoder in Large Language Models: How Attention Masks, KV Cache Serving, and Scaling Dynamics Decided the Architecture Race

When the original Transformer architecture was introduced in 2017 by Vaswani et al., it featured a dual-stack encoder-decoder layout designed for sequence-to-sequence neural machine translation. Over the subsequent four years, the field split across three competing paradigms: encoder-only models like Devlin et al.'s BERT for understanding, encoder-decoder models like Raffel et al.'s T5 and Lewis et al.'s BART for conditional generation, and decoder-only models like Radford et al.'s GPT series for autoregressive generation.

Today, nearly every frontier and open-weight foundation model, from LLaMA and Mistral to GPT-4, Qwen, and DeepSeek, relies exclusively on the decoder-only architecture. This convergence was not accidental. It was driven by three decisive factors: pre-training token density, key-value (KV) cache serving simplicity, and the mechanics of in-context learning.

The Three Architectural Paradigms

To understand why decoder-only architectures dominated, it is necessary to examine how attention masks and sequence processing differ across the three structural layouts.

1. Encoder-Only (BERT, RoBERTa)
Input: [Token 1] <---> [Token 2] <---> [Token 3] <---> [Token 4]
Mask: Full Bidirectional (All tokens attend to all tokens)

2. Encoder-Decoder (T5, BART, Original Transformer)
Encoder: [Token 1] <---> [Token 2] (Bidirectional)
                   |           |   (Cross-Attention)
                   v           v
Decoder: [Target 1] ---> [Target 2] (Causal Auto-regressive)

3. Decoder-Only (GPT, LLaMA, Mistral, DeepSeek)
Input + Output: [Prompt 1] ---> [Prompt 2] ---> [Token 1] ---> [Token 2]
Mask: Lower-Triangular Causal Mask (Tokens only attend to past positions)

In standard scaled dot-product attention, the attention weights are computed as:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + M\right)V$$

The matrix $M$ is the attention mask that defines which tokens can exchange information with one another.

• Encoder-Only Models: Set $M_{ij} = 0$ for all token positions $i$ and $j$. Every token attends bidirectionally to every other token. While this produces rich contextual representations for classification or sequence labeling, encoder-only models cannot generate coherent autoregressive text without computationally inefficient iterative sampling.
• Encoder-Decoder Models: Use two distinct sub-networks. The encoder runs bidirectional self-attention ($M_{ij} = 0$). The decoder applies causal self-attention ($M_{ij} = -\infty$ for $j > i$) to prevent looking ahead, combined with cross-attention layers where decoder query vectors $Q_{\text{dec}}$ attend over the final encoder key-value representations ($K_{\text{enc}}, V_{\text{enc}}$).
• Decoder-Only Models: Consist of a single stack of identical transformer blocks governed by a lower-triangular causal attention mask ($M_{ij} = -\infty$ for $j > i$). Every token can attend only to preceding tokens and itself.

A hybrid variant known as Prefix LM (or non-causal decoder) allows bidirectional attention across the input prompt ($1 \le i, j \le k$), while enforcing causal masking for all newly generated continuation tokens ($i > k$).

Pre-Training Dynamics: Supervision Density and Gradient Flow

One of the primary advantages of decoder-only architectures lies in pre-training sample efficiency.

In masked language modeling (MLM) and span-corruption objectives (used by BERT, T5, and BART), the model masks a subset of tokens, typically 15%, and attempts to reconstruct them:

$$\mathcal{L}_{\text{span}} = -\sum_{i \in \text{masked}} \log P(x_i \mid x_{\setminus \text{masked}})$$

Under this regime, 85% of the input tokens in any given training sequence produce no direct prediction loss. While bidirectional self-attention allows the model to build contextual representations of the unmasked tokens, the gradient signal per forward-backward pass is sparse.

In contrast, autoregressive causal language modeling (CLM) computes the cross-entropy loss at every position across the entire sequence length $L$:

$$\mathcal{L}_{\text{CLM}} = -\sum_{t=1}^{L} \log P(x_t \mid x_{<t})$$

Every single token acts simultaneously as a target for the preceding prefix and as context for all subsequent tokens. For a sequence of 4,096 tokens, a causal decoder computes 4,095 next-token loss gradients in a single forward-backward pass.

Large-scale empirical evaluations by Wang et al. (2022) demonstrated that when models are trained under purely unsupervised pre-training budgets, causal decoder-only models systematically outperform encoder-decoder and prefix-LM architectures on zero-shot and few-shot benchmarks.

While encoder-decoder models can achieve strong performance when pre-training is followed by extensive supervised multitask fine-tuning (as demonstrated by Tay et al. in the UL2 study), the raw scaling efficiency of next-token prediction across multi-trillion-token corpora made decoder-only models the compute-optimal choice.

Inference Serving Economics and the KV Cache Bottleneck

In production serving systems, the architectural choice dictates memory allocation, paging efficiency, and compute utilization.

During autoregressive generation, computing attention over previously generated tokens without re-evaluating the entire sequence requires caching key and value vectors in high-bandwidth memory (HBM).

The Decoder-Only Serving Pipeline

In a decoder-only model:

1. Prefill Phase: The entire input prompt of length $L_{\text{prompt}}$ is processed in parallel across all layers in a single forward pass, populating the initial KV cache.
2. Decode Phase: Each step generates a single new token $x_t$. The new token computes query, key, and value vectors ($q_t, k_t, v_t$). The new $(k_t, v_t)$ are appended to the existing cache, and attention is computed across the accumulated history:

$$\text{Attention}(q_t, K_{1:t}, V_{1:t}) = \text{softmax}\left(\frac{q_t K_{1:t}^T}{\sqrt{d_k}}\right)V_{1:t}$$

This architecture provides structural uniformity:

• Every transformer layer maintains an identical KV cache shape: [batch_size, num_kv_heads, seq_len, head_dim].
• Memory allocators like PagedAttention in vLLM manage a single pool of non-contiguous physical memory blocks.
• Prefix caching engines (such as SGLang's RadixAttention) can directly match and reuse causal KV cache prefixes across multi-turn user requests.

The Encoder-Decoder Serving Pipeline

In contrast, encoder-decoder serving introduces dual-state memory fragmentation:

1. The encoder processes the prompt $L_{\text{enc}}$ and writes static encoder KV caches ($K_{\text{enc}}, V_{\text{enc}}$) across all layers.
2. The decoder maintains a separate, dynamically expanding causal self-attention KV cache ($K_{\text{dec}}, V_{\text{dec}}$).
3. At every generation step, each decoder layer executes two attention operations: causal self-attention over $(K_{\text{dec}}, V_{\text{dec}})$ and cross-attention over $(K_{\text{enc}}, V_{\text{enc}})$.

This dual cache structure complicates memory paging, increases kernel launch overhead, and makes dynamic prefix caching across multi-turn dialogues significantly more difficult to manage.

Compute Allocation Efficiency

Parameter utilization during generation also favors decoder-only architectures:

• In an encoder-decoder model with $N$ total parameters ($N/2$ in the encoder and $N/2$ in the decoder), the encoder parameters sit completely idle during the entire token generation phase.
• In an $N$-parameter decoder-only model, 100% of the parameter capacity is actively engaged during both prompt ingestion and every subsequent generation step.

In-Context Learning and the Unified Text Interface

The emergence of in-context learning (ICL), documented by Radford et al. in GPT-2 and Brown et al. in GPT-3, redefined how foundation models interact with users.

In-context learning frames every task as an unbroken sequence of tokens:

[System Prompt] -> [Few-Shot Example 1] -> [Few-Shot Example 2] -> [User Input] -> [Generated Answer]

Decoder-only architectures treat this entire sequence as a single autoregressive trajectory. There is no artificial architectural boundary separating "inputs" from "outputs". This uniformity enables:

1. Arbitrary Multi-Turn Conversations: User turns and assistant turns simply append to the running causal context without requiring re-encoding or structural re-routing.
2. Chain-of-Thought and Deliberation: Intermediate reasoning steps, scratchpads, and thinking tokens flow directly into the causal context, allowing subsequent reasoning tokens to condition on earlier intermediate steps.
3. Agentic Tool Calling: Tool execution results, environment observations, and API responses are appended sequentially to the context, maintaining full historical state within a unified KV cache.

Encoder-decoder models impose a rigid structural split. The user must decide which portion of the prompt belongs in the encoder and where the decoder generation should begin. For multi-step reasoning or agentic loops, the encoder input must either be repeatedly re-encoded or manually synchronized with decoder outputs, breaking the simplicity of the single-stream token interface.

Architectural Comparison Across Key Dimensions

• Attention Mask Structure:
• Encoder-Only (BERT): Fully bidirectional across all token positions.
• Encoder-Decoder (T5, BART): Bidirectional self-attention in the encoder; causal self-attention and cross-attention in the decoder.
• Prefix LM (UL2): Bidirectional attention within the prefix prompt; causal masking across continuation tokens.
• Decoder-Only (LLaMA, GPT): Strictly lower-triangular causal attention across the entire sequence.
• Pre-Training Loss and Supervision Density:
• Encoder-Only: Masked language modeling (typically 15% of tokens masked). Supervision density is low (0.15 loss per token).
• Encoder-Decoder: Span corruption and denoising (typically 15% of tokens corrupted). Supervision density is low (0.15 loss per token).
• Prefix LM: Mixed span corruption and prefix-causal language modeling. Supervision density is moderate to high.
• Decoder-Only: Autoregressive next-token prediction across all sequence positions. Supervision density is maximal (1.0 loss per token).
• KV Cache and Serving Architecture:
• Encoder-Only: Not applicable (non-generative).
• Encoder-Decoder: Dual-state memory required (static encoder KV cache plus expanding dynamic decoder KV cache).
• Prefix LM: Unified memory buffer with mixed bidirectional and causal attention kernels.
• Decoder-Only: Unified single-buffer dynamic KV cache per layer, highly optimized for PagedAttention and prefix caching.
• Parameter Utilization During Generation:
• Encoder-Only: N/A.
• Encoder-Decoder: Approximately 50% of total parameters (the encoder stack) sit idle during token decoding.
• Prefix LM: 100% of model parameters participate in both prefill and decoding phases.
• Decoder-Only: 100% of model parameters participate in both prefill and decoding phases.
• In-Context Learning and Agentic Execution:
• Encoder-Only: Unsupported for open-ended generation.
• Encoder-Decoder: Rigid structural boundary requiring separation of input context and output generation.
• Prefix LM: Supported across multi-task prompt formats.
• Decoder-Only: Fully natural unbroken 1D token timeline, supporting multi-turn dialogue, tool calls, and chain-of-thought scratchpads.

Where Encoder-Decoder Still Holds Advantages

While decoder-only architectures dominate general language modeling, encoder-decoder models retain clear strengths in dedicated sequence-to-sequence domains:

• Speech Recognition and Audio Processing: Models like OpenAI's Whisper use an audio encoder (processing mel-spectrograms bidirectionally) paired with an autoregressive text decoder. The separation allows specialized acoustic feature extraction before generating linguistic tokens.
• Dedicated Machine Translation: When translating between fixed language pairs, an encoder-decoder architecture can build bidirectional source representations while producing concise target sequences.
• Multimodal Encoders: Modern vision-language architectures frequently employ bidirectional vision encoders (such as ViT or SigLIP) to extract visual patches, projecting those representations directly into the input space of a causal decoder-only language model.

Summary

The triumph of decoder-only transformers was not a matter of aesthetic preference. Causal autoregression maximizes supervision density during pre-training by computing loss across every sequence token, eliminates parameter idling during inference, streamlines KV cache management in distributed serving clusters, and provides a natural, unbroken substrate for in-context learning and multi-step agent reasoning.

Sources

• Vaswani, A., et al. (2017). Attention Is All You Need. arXiv:1706.03762
• Radford, A., et al. (2018). Improving Language Understanding by Generative Pre-Training. OpenAI Paper
• Radford, A., et al. (2019). Language Models are Unsupervised Multitask Learners. OpenAI Paper
• Devlin, J., et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805
• Raffel, C., et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683
• Lewis, M., et al. (2020). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. arXiv:1910.13461
• Wang, T., et al. (2022). What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?. arXiv:2204.05832
• Tay, Y., et al. (2022). UL2: Unifying Language Learning Paradigms. arXiv:2205.05131
• Brown, T., et al. (2020). Language Models are Few-Shot Learners. arXiv:2005.14165
• Touvron, H., et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971