Transformers47 articles

Transformers

Articles

State Space Models and Mamba (Mamba-1 and Mamba-2): Mathematical Foundations, Selective State Spaces, Structured State Space Duality (SSD), and Linear-Time Sequence Modeling
State Space Models (SSMs) and their modern selective formulations, most notably Mamba-1 and Mamba-2, represent a foundational alternative to the standard Transformer architecture for sequence modeling. While multi-head self-attention scales quadratically with sequence length ($O(T^2)$) and requires an ever-expanding Key-Value (KV) cache during autoregressive generation ($O(T)$), State Space Models achieve linear time complexity ($O(T)$) during training and constant memory footprint ($O(1)$) per
August 28, 20261 min
FlashAttention (FlashAttention-1, 2, and 3): Mathematical Foundations, IO-Awareness, Online Softmax Tiling, and Asynchronous Hardware Acceleration
The standard scaled dot-product attention mechanism defines the computational core of modern autoregressive large language models and vision transformers. While algorithmically straightforward, its practical execution on modern graphics processing units (GPUs) has historically suffered from severe hardware memory bottlenecks. Standard self-attention incurs an $O(N^2)$ asymptotic computational and memory footprint with respect to sequence length $N$. On modern accelerators like NVIDIA A100 and H
August 28, 20261 min
Mixture-of-Depths (MoD): Mathematical Foundations, Dynamic Token-Level Compute Routing, Top-k Capacity Budgeting, and FLOP-Optimal Transformer Architectures
Standard autoregressive Transformers allocate an identical compute budget to every token in a sequence. Regardless of whether a token represents a trivial punctuation mark, a common grammatical function word, or a complex semantic reasoning step, the model applies the exact same sequence of multi-head self-attention and multilayer perceptron (MLP) operations across all $L$ layers. In Mixture-of-Depths: Dynamically allocating compute in transformer-based language models, researchers at Google De
August 28, 20261 min
FlashAttention: IO-Aware Exact Attention, Tiling, Online Softmax, and the Evolution to FlashAttention-3
FlashAttention: IO-Aware Exact Attention, Tiling, Online Softmax, and the Evolution to FlashAttention-3 The attention mechanism is the computational bottleneck of every Transformer model. Standard implementations materialize the full $N \times N$ attention matrix in high-bandwidth memory (HBM), incurring $O(N^2)$ memory reads and writes that dominate runtime long before arithmetic intensity saturates the GPU. FlashAttention and its successors eliminate this bottleneck by restructuring the atten
August 28, 20261 min
Activation Checkpointing and Selective Rematerialization: Mathematical Foundations, Sub-Linear Memory Scaling, Recomputation Trade-Offs, and Megatron-LM Attention Partitioning
Activation Checkpointing and Selective Rematerialization: Mathematical Foundations, Sub-Linear Memory Scaling, Recomputation Trade-Offs, and Megatron-LM Attention Partitioning Training modern large language models requires orchestrating hundreds of billions of parameters across distributed GPU clusters. While parameter counts and optimizer states are fixed for a given model architecture, the activation memory generated during forward passes scales directly with sequence length, batch size, and
August 28, 20261 min
RMSNorm and SwiGLU: Mathematical Foundations of Scaling-Invariant Normalization, Gated Activations, and FFN Architectures in Modern LLMs
The architectural baseline of modern autoregressive large language models has converged on a distinct set of mathematical primitives. While early Transformer architectures relied on standard Layer Normalization, Post-LN residual routing, and two-layer Multi-Layer Perceptrons with ReLU or GELU activations, state-of-the-art open-weights models such as LLaMA, Mistral, Gemma, Qwen, and DeepSeek utilize a different combination: Root Mean Square Layer Normalization (RMSNorm), Pre-LN residual connectio
August 27, 20261 min
Tensor Parallelism and Pipeline Parallelism: Mathematical Foundations of Megatron-LM 1D Slicing, Sequence Parallelism, and 1F1B Scheduling
Scaling modern large language models beyond tens of billions of parameters quickly exhausts the physical memory and compute throughput of individual graphics processing units. A 70-billion-parameter model stored in 16-bit floating-point (FP16 or BF16) requires 140 GB of VRAM solely for model weights. During full-precision training with the Adam optimizer, parameter states, gradients, and optimizer momentum terms demand approximately 16 to 18 bytes per parameter (amounting to 1.12 TB to 1.26 TB f
August 27, 20261 min
RingAttention and Context Parallelism: Mathematical Foundations, Distributed Blockwise Attention, Circular Communication Topologies, and Million-Token Context Scaling
RingAttention and Context Parallelism: Mathematical Foundations, Distributed Blockwise Attention, Circular Communication Topologies, and Million-Token Context Scaling Standard Transformer architectures face a quadratic memory and computational barrier in their self-attention mechanism. While FlashAttention solved the high-bandwidth memory (HBM) IO bottleneck on single devices by tiling matrices within SRAM, scaling sequence lengths beyond hundreds of thousands or millions of tokens quickly exce
August 27, 20261 min
StreamingLLM and Attention Sinks: Mathematical Foundations of Softmax Normalization Artifacts, Initial Token Anchoring, and Infinite-Context Rolling KV-Cache Mechanics
StreamingLLM and Attention Sinks: Mathematical Foundations of Softmax Normalization Artifacts, Initial Token Anchoring, and Infinite-Context Rolling KV-Cache Mechanics In autoregressive Large Language Models (LLMs), long-context deployment is constrained by the quadratic compute complexity of attention and the linear memory growth of the Key-Value (KV) cache. For an LLM processing a sequence of length $L$ with batch size $b$, hidden dimension $d$, and $n_{kv}$ key-value heads across $N_L$ trans
August 27, 20261 min
FlashAttention: Mathematical Foundations, IO-Aware Tiling, Online Softmax Scaling, and Memory-Hierarchy Optimization in Transformer Architectures
Standard multi-head self-attention represents the core computational bottleneck in scaling modern Transformer models to long context windows. While the theoretical arithmetic operations of self-attention scale quadratically with sequence length $O(N^2)$, modern GPU execution profiles reveal that standard implementations are constrained not by floating-point arithmetic throughput (FLOPs), but by memory access latency and bandwidth between High Bandwidth Memory (HBM) and on-chip Static Random-Acce
August 27, 20261 min
Rotary Position Embedding (RoPE): Mathematical Foundations, Complex Plane Rotation, Relative Distance Decay, and Context Extension Mechanics
Transformer architectures lack inherent awareness of token order due to the permutation equivalence of standard self-attention operations. Early sequence models addressed this structural limitation through absolute additive positional embeddings or learned lookup tables. However, additive positional encodings do not naturally capture relative token distances and degrade severely when evaluating sequence lengths beyond the training horizon. RoFormer: Enhanced Transformer with Rotary Position Emb
August 26, 20261 min
Transformers as Support Vector Machines: Mathematical Foundations of Implicit Bias, Margin Maximization, and Hard-Attention Convergence
Understanding why overparameterized deep neural networks generalize well despite possessing sufficient capacity to memorize training data remains a fundamental question in theoretical machine learning. In classical linear models trained on separable data under exponential-tailed losses (such as logistic or cross-entropy loss), gradient descent exhibits an implicit bias: parameter iterates diverge in norm while their directions asymptotically converge to the maximum-margin hyperplane, known as th
August 26, 20261 min
GPTQ: Mathematical Foundations, Optimal Brain Surgeon Inversion, and Second-Order Error Minimization in LLM Quantization
Large language model inference during autoregressive generation is overwhelmingly memory-bandwidth bound. For batch size 1 decoding, each generated token requires streaming every parameter of a model from High Bandwidth Memory (HBM) into GPU SRAM and Tensor Cores. A 70-billion parameter model in 16-bit precision (FP16 or BF16) requires roughly 140 GB of VRAM, exceeding the capacity of a single 80 GB NVIDIA A100 or H100 GPU and demanding multi-GPU tensor parallelism solely to hold the model weigh
August 26, 20261 min
Mixture-of-Depths: Mathematical Foundations, Dynamic Compute Routing, Capacity-Constrained Tensors, and IsoFLOP Scaling
Mixture-of-Depths (MoD): Mathematical Foundations, Dynamic Compute Routing, Capacity-Constrained Tensors, and IsoFLOP Scaling In standard autoregressive Transformer architectures, computational effort is distributed uniformly across all tokens in a sequence. Every token position $i \in \{1, \dots, S\}$ passes through every layer $l \in \{1, \dots, L\}$, executing identical matrix multiplications across multi-head self-attention and feed-forward networks (FFN). This architectural constraint igno
August 26, 20261 min
Induction Heads: Mathematical Foundations, Two-Layer Circuit Composition, and the Emergence of In-Context Learning in Transformers
Induction Heads: Mathematical Foundations, Two-Layer Circuit Composition, and the Emergence of In-Context Learning in Transformers One of the defining capabilities of modern autoregressive large language models is in-context learning: the ability to infer rules, adapt to task formats, and execute complex few-shot instructions purely from prompt context without updating neural network weights. For years, in-context learning was treated as an enigmatic, emergent property of large-scale autoregres
August 25, 20261 min
FlashAttention: Mathematical Foundations, IO-Aware Tiling, Online Softmax, and Recomputation Dynamics
Standard self-attention in the Transformer architecture scales quadratically with sequence length in both time and memory complexity. While algorithmic research historically focused on reducing FLOP counts via sparse or low-rank approximations, Tri Dao et al. (2022) demonstrated that the practical wall-clock bottleneck in multi-head attention is not compute capability, but memory IO. FlashAttention reformulates exact scaled dot-product attention as an IO-aware algorithm. By leveraging the GPU m
August 25, 20261 min
Integrated Gradients: How Axiomatic Attribution Solves the Gradients-at-Saturation Problem in Deep Neural Networks
Integrated Gradients: How Axiomatic Attribution Solves the Gradients-at-Saturation Problem in Deep Neural Networks Feature attribution methods in deep learning aim to answer a fundamental interpretability question: given an input vector and a trained neural network, how much did each input dimension contribute to the model's final output score? In natural language processing and computer vision, practitioners routinely need to identify which input tokens, pixels, or tabular variables drove a sp
August 24, 20261 min
Relative Positional Encodings in Transformers: How Shaw's Attention, Transformer-XL, and T5 Relative Biases Preserve Translation Invariance
Relative Positional Encodings in Transformers: How Shaw's Attention, Transformer-XL, and T5 Relative Biases Preserve Translation Invariance Standard self-attention operations in transformer architectures possess no inherent awareness of sequence order. Because scaled dot-product attention computes interactions across sets of tokens without regard to index ordering, early models relied on absolute positional encodings to inject sequential structure. While absolute encodings assign rigid coordina
August 24, 20261 min
RWKV Architecture: How Receptance Weighted Key Value Decay Combines RNN Efficiency with Transformer Parallelizability
The dominant paradigm in natural language processing relies on the Transformer architecture, which calculates scaled dot-product self-attention across all token pairs in a sequence. While self-attention provides strong in-context retrieval and representation capacity, it imposes quadratic computational and memory complexity, scaling as O(N^2) with sequence length N during training and generating a continuously expanding Key-Value (KV) cache during autoregressive inference. Traditional Recurrent
August 24, 20261 min
The Platonic Representation Hypothesis: How Foundation Models Across Modalities Converge on a Shared Geometry of Reality
In 375 BC, Plato introduced the Allegory of the Cave, describing prisoners who could only perceive reality through shadows projected onto a stone wall. In modern machine learning, deep neural networks ingest distinct sensory shadows of the physical world: vision models process pixel arrays, large language models ingest tokenized text, and speech models parse raw acoustic waveforms. Historically, the machine learning literature assumed that models trained on different modalities, objectives, and
August 23, 20261 min