Transformer8 articles

Transformer

Articles

Multi-Head Latent Attention (MLA): Mathematical Foundations, Low-Rank KV Compression, Decoupled Rotary Positional Embeddings, and Inference Matrix Absorption
Autoregressive transformer inference in production is fundamentally bounded by memory bandwidth rather than compute capability during the decoding phase. As context lengths scale from 8,000 to 128,000 tokens and beyond, storing and loading the Key-Value (KV) cache across tens of attention layers consumes the vast majority of GPU High-Bandwidth Memory (HBM) and degrades serving throughput. While earlier architectures mitigated this bottleneck via Multi-Query Attention (MQA) or Grouped-Query Atte
August 28, 20261 min
Grouped-Query Attention (GQA) and Multi-Query Attention (MQA): Mathematical Foundations, Arithmetic Intensity, and KV-Cache Scaling Mechanics
Grouped-Query Attention (GQA) and Multi-Query Attention (MQA): Mathematical Foundations, Arithmetic Intensity, and KV-Cache Scaling Mechanics Autoregressive large language model inference is governed by a fundamental hardware asymmetry. During the initial prompt processing phase (prefill), computation is compute-bound because matrix multiplications process entire sequences in parallel. During sequential token generation (decoding), execution becomes strictly memory-bandwidth bound. For each gen
August 27, 20261 min
Grouped-Query Attention (GQA) and Multi-Query Attention (MQA): Mathematical Foundations, KV Cache Memory Reduction, and Arithmetic Intensity in Transformer Serving
Autoregressive generation in large language models exhibits an asymmetric computational profile between initial prompt processing (prefill) and subsequent incremental generation (decode). While prefill parallelizes across all prompt tokens and achieves high compute utilization on modern tensor accelerators, token-by-token decoding is heavily memory-bandwidth bound. Each generated token requires reading the entire accumulated Key-Value (KV) cache from High Bandwidth Memory (HBM) into on-chip SRAM
August 25, 20261 min
Multi-Head Latent Attention: Mathematical Foundations, Low-Rank KV Compression, and Decoupled RoPE in Transformer Architectures
Multi-Head Attention (MHA) has served as the core sequence-mixing primitive in autoregressive Transformer architectures since the introduction of the Transformer by Vaswani et al. (2017). In production serving environments, autoregressive generation requires caching key and value projections for all previous tokens in high-bandwidth GPU memory (HBM). As sequence lengths expand toward 128k tokens and beyond, this Key-Value (KV) cache grows linearly with context length, batch size, and layer count
August 25, 20261 min
Cross-Layer Attention: How Sharing Key-Value Tensors Across Transformer Layers Slashes KV Cache Memory
In autoregressive large language models, the primary operational ceiling for serving long sequences and high batch concurrency is the Key-Value (KV) cache. During standard generation, every transformer layer computes and stores key and value activations for every token in the sequence to prevent quadratic recomputation during subsequent autoregressive decoding steps. While this mechanism transforms inference time complexity from $O(N^2)$ to $O(N)$ per generated token, it introduces a massive me
August 21, 20261 min
Multi-Head Attention in Large Language Models: How Query, Key, and Value Projections Route Information Across Subspaces
Multi-Head Attention in Large Language Models: How Query, Key, and Value Projections Route Information Across Subspaces The attention mechanism serves as the computational core of modern Transformer architectures, providing a mechanism for tokens to dynamically exchange information across arbitrary sequence positions without recurrent state transitions or fixed convolutional receptive fields. While single-head attention computes a single set of attention weights per token pair, modern large lan
August 20, 20261 min
Sliding Window Attention in Large Language Models: How Bounded Receptive Fields, Interleaved Layers, and Rolling KV Buffers Scale Contexts
Standard causal multi-head attention imposes two severe computational constraints as sequence lengths expand into tens or hundreds of thousands of tokens. First, calculating pairwise query-key dot products scales quadratically with sequence length, requiring $O(N^2)$ floating-point operations. Second, autoregressive generation requires caching key and value projections for all preceding tokens, causing the key-value (KV) cache to grow linearly with sequence length $O(N)$ across all layers and at
August 20, 20261 min
Attention with Linear Biases (ALiBi): How Static Positional Slopes Enable Zero-Shot Context Extrapolation
Large language models process sequences by transforming discrete tokens into continuous vector representations. Standard dot-product self-attention is permutation-invariant: without explicit positional information, the attention operation treats a sequence as an unordered bag of tokens. Early transformer architectures addressed this limitation using Absolute Positional Embeddings (APE), either through fixed sinusoidal functions or learned lookup tables added directly to token embeddings. While
August 20, 20261 min