Deep Learning58 articles

Deep Learning

Articles

Mixture-of-Experts (MoE) Routing and Load Balancing: Mathematical Foundations, Top-k Gating, Auxiliary Losses, Auxiliary-Loss-Free Balancing, and Distributed Expert Parallelism
Scaling autoregressive language models under fixed inference latency and training compute budgets has forced a departure from purely dense feed-forward architectures. In a standard dense transformer, every parameter is activated for every token, resulting in computational complexity and memory bandwidth consumption that scale linearly with the total parameter count. Mixture-of-Experts (MoE) architectures decouple parameter capacity from per-token compute by replacing monolithic Multi-Layer Perc
August 28, 20261 min
FlashAttention (FlashAttention-1, 2, and 3): Mathematical Foundations, IO-Awareness, Online Softmax Tiling, and Asynchronous Hardware Acceleration
The standard scaled dot-product attention mechanism defines the computational core of modern autoregressive large language models and vision transformers. While algorithmically straightforward, its practical execution on modern graphics processing units (GPUs) has historically suffered from severe hardware memory bottlenecks. Standard self-attention incurs an $O(N^2)$ asymptotic computational and memory footprint with respect to sequence length $N$. On modern accelerators like NVIDIA A100 and H
August 28, 20261 min
Mixture-of-Depths (MoD): Mathematical Foundations, Dynamic Token-Level Compute Routing, Top-k Capacity Budgeting, and FLOP-Optimal Transformer Architectures
Standard autoregressive Transformers allocate an identical compute budget to every token in a sequence. Regardless of whether a token represents a trivial punctuation mark, a common grammatical function word, or a complex semantic reasoning step, the model applies the exact same sequence of multi-head self-attention and multilayer perceptron (MLP) operations across all $L$ layers. In Mixture-of-Depths: Dynamically allocating compute in transformer-based language models, researchers at Google De
August 28, 20261 min
Zero Redundancy Optimizer (ZeRO) and Fully Sharded Data Parallelism (FSDP): Mathematical Foundations of Memory Partitioning, Communication Complexity, and Overlapped Execution
Training modern frontier large language models requires orchestrating compute across thousands of accelerators. As parameter counts scaled from billions to hundreds of billions, the memory capacity of single accelerators (such as 80 GB on an NVIDIA H100 or 96 GB on an H200) became a primary scaling bottleneck. Traditional distributed training paradigms relied on standard Data Parallelism (DDP), where every GPU maintains a complete replica of the model weights, gradients, and optimizer states, sy
August 27, 20261 min
RMSNorm and SwiGLU: Mathematical Foundations of Scaling-Invariant Normalization, Gated Activations, and FFN Architectures in Modern LLMs
The architectural baseline of modern autoregressive large language models has converged on a distinct set of mathematical primitives. While early Transformer architectures relied on standard Layer Normalization, Post-LN residual routing, and two-layer Multi-Layer Perceptrons with ReLU or GELU activations, state-of-the-art open-weights models such as LLaMA, Mistral, Gemma, Qwen, and DeepSeek utilize a different combination: Root Mean Square Layer Normalization (RMSNorm), Pre-LN residual connectio
August 27, 20261 min
Mixture-of-Experts (MoE) Routing and Load Balancing: Mathematical Foundations of Noisy Top-k Gating, Expert Capacity, Auxiliary Balancing Losses, and Loss-Free Load Balancing
Mixture-of-Experts (MoE) Routing and Load Balancing: Mathematical Foundations of Noisy Top-k Gating, Expert Capacity, Auxiliary Balancing Losses, and Loss-Free Load Balancing Scaling dense transformer models incurs computational costs that grow proportionally with parameter count. In dense architectures, every token in a sequence activates all feed-forward network (FFN) parameters across every layer. Sparse Mixture-of-Experts (MoE) architectures decouple parameter count from per-token compute b
August 27, 20261 min
StreamingLLM and Attention Sinks: Mathematical Foundations of Softmax Normalization Artifacts, Initial Token Anchoring, and Infinite-Context Rolling KV-Cache Mechanics
StreamingLLM and Attention Sinks: Mathematical Foundations of Softmax Normalization Artifacts, Initial Token Anchoring, and Infinite-Context Rolling KV-Cache Mechanics In autoregressive Large Language Models (LLMs), long-context deployment is constrained by the quadratic compute complexity of attention and the linear memory growth of the Key-Value (KV) cache. For an LLM processing a sequence of length $L$ with batch size $b$, hidden dimension $d$, and $n_{kv}$ key-value heads across $N_L$ trans
August 27, 20261 min
Multi-Head Latent Attention (MLA): Mathematical Foundations, Low-Rank KV Compression, Decoupled RoPE Mechanics, and Matrix Absorption
Multi-Head Latent Attention (MLA): Mathematical Foundations, Low-Rank KV Compression, Decoupled RoPE Mechanics, and Matrix Absorption Autoregressive inference in large language models is fundamentally constrained by memory bandwidth rather than floating-point computation throughput. During the generation phase, the transformer must load the Key-Value (KV) cache of all prior tokens from high-bandwidth memory (HBM) to on-chip SRAM for every newly generated token. As context lengths scale to hundr
August 27, 20261 min
Post-Training Quantization (PTQ): Mathematical Foundations of Optimal Brain Surgeon, GPTQ Hessian Inversion, AWQ Salient Scaling, and SmoothQuant Outlier Migration
Post-Training Quantization (PTQ): Mathematical Foundations of Optimal Brain Surgeon, GPTQ Hessian Inversion, AWQ Salient Scaling, and SmoothQuant Outlier Migration Serving modern large language models at scale requires addressing severe hardware constraints. In autoregressive generation, decoding is fundamentally bounded by memory bandwidth rather than floating-point computation throughput. Each generated token requires reading every parameter from High Bandwidth Memory (HBM) into SRAM to perfo
August 27, 20261 min
Low-Rank Adaptation (LoRA) and QLoRA: Mathematical Foundations, Intrinsic Rank Dynamics, NF4 Quantization, and Parameter-Efficient Fine-Tuning
Full fine-tuning of large language models requires updating every parameter matrix across all transformer blocks. In production architectures spanning tens to hundreds of billions of parameters, the computational and memory footprint of updating billions of weights with first-order and second-order optimizer states becomes prohibitive. Low-Rank Adaptation (LoRA) and its quantized counterpart QLoRA provide mathematically grounded parameter-efficient fine-tuning (PEFT) frameworks. By decomposing
August 26, 20261 min
FlashAttention: Mathematical Foundations, Online Softmax Tiling, IO-Awareness, and Exact Attention Scaling
Standard multi-head self-attention in the Transformer architecture exhibits quadratic time and memory complexity with respect to sequence length $N$. While the $O(N^2)$ computational complexity is widely cited, the primary performance bottleneck in production hardware is not arithmetic throughput, but memory access overhead. On modern GPU architectures such as NVIDIA A100 and H100, tensor processing cores execute matrix multiplications at teraflop and petaflop scales, but memory bandwidth betwee
August 26, 20261 min
Activation-Aware Weight Quantization (AWQ): Mathematical Foundations, Salient Weight Protection, and Hardware-Efficient Low-Bit Inference
Large language model inference during autoregressive generation is overwhelmingly memory bandwidth bound. While the prefill phase processes multiple prompt tokens in parallel with high arithmetic intensity, the token generation phase computes matrix-vector multiplications ($M=1$) for each sequential token. In this regime, the GPU spends the vast majority of its cycle budget streaming model parameters from High Bandwidth Memory (HBM) or GDDR into SRAM rather than performing floating-point arithme
August 26, 20261 min
Mixture-of-Depths: Mathematical Foundations, Dynamic Compute Routing, Capacity-Constrained Tensors, and IsoFLOP Scaling
Mixture-of-Depths (MoD): Mathematical Foundations, Dynamic Compute Routing, Capacity-Constrained Tensors, and IsoFLOP Scaling In standard autoregressive Transformer architectures, computational effort is distributed uniformly across all tokens in a sequence. Every token position $i \in \{1, \dots, S\}$ passes through every layer $l \in \{1, \dots, L\}$, executing identical matrix multiplications across multi-head self-attention and feed-forward networks (FFN). This architectural constraint igno
August 26, 20261 min
Knowledge Distillation: Mathematical Foundations, Dark Knowledge, Soft Target Regularization, and Sequence-Level Policy Transfer
Knowledge Distillation: Mathematical Foundations, Dark Knowledge, Soft Target Regularization, and Sequence-Level Policy Transfer Knowledge distillation is a foundational model compression and transfer technique wherein a compact "student" neural network is trained to reproduce the functional behavior, internal representations, or output distributions of a larger, high-capacity "teacher" model or ensemble. First formalized in modern deep learning by Hinton, Vinyals, and Dean (2015), following ea
August 25, 20261 min
BitNet b1.58: Mathematical Foundations of 1.58-Bit Ternary Quantization, Absmean Scaling, and Multiplication-Free LLM Architectures
Standard Large Language Models (LLMs) rely on 16-bit floating-point (FP16 or BF16) matrix multiplications for both pre-training and autoregressive inference. While post-training quantization techniques (such as GPTQ, AWQ, and dynamic FP8/INT4 scaling) compress weights after training, they often introduce perplexity degradation at ultra-low bitwidths ($<4$ bits) or still require floating-point dequantization at runtime. Quantization-aware pre-training architectures directly tackle this computati
August 25, 20261 min
The Muon Optimizer: Mathematical Foundations, Newton-Schulz Polar Iterations, and Spectral Steepest Descent in LLM Pre-Training
Transformer pre-training has long relied on AdamW as the default first-order optimizer. While AdamW provides coordinate-wise scale invariance through running estimates of first and second gradient moments, it treats every weight parameter as an isolated scalar. For modern neural networks composed primarily of two-dimensional linear projections, attention transformations, and feed-forward weight matrices, this coordinate-wise factorization neglects the structural transformation properties of matr
August 25, 20261 min
Decoupled Weight Decay Regularization (AdamW): Mathematical Foundations, L2 Regularization Failure in Adaptive Gradient Methods, and Hyperparameter Dynamics
Decoupled weight decay regularization, formalized by Ilya Loshchilov and Frank Hutter in 2017 and published at ICLR 2019, resolved a fundamental flaw in how adaptive gradient algorithms like Adam implemented weight regularization. Prior to AdamW, deep learning frameworks implemented weight decay as L2 regularization by adding the gradient of the squared L2 norm directly to the loss gradient. While mathematically identical for standard stochastic gradient descent (SGD), this formulation breaks do
August 25, 20261 min
FlashAttention: Mathematical Foundations, IO-Aware Tiling, Online Softmax, and Recomputation Dynamics
Standard self-attention in the Transformer architecture scales quadratically with sequence length in both time and memory complexity. While algorithmic research historically focused on reducing FLOP counts via sparse or low-rank approximations, Tri Dao et al. (2022) demonstrated that the practical wall-clock bottleneck in multi-head attention is not compute capability, but memory IO. FlashAttention reformulates exact scaled dot-product attention as an IO-aware algorithm. By leveraging the GPU m
August 25, 20261 min
Batch Normalization: Mathematical Foundations, Gradient Smoothing Dynamics, and Why Sequence Models Adopted Layer Normalization
Batch Normalization remains one of the most widely implemented algorithmic developments in the history of deep learning. Introduced by Sergey Ioffe and Christian Szegedy in their 2015 paper, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, the technique enabled stable training of deep feedforward networks and convolutional architectures at significantly higher learning rates. While initially designed for computer vision architectures such as ResNet a
August 25, 20261 min
Curriculum Learning in Large Language Models: How Difficulty Pacing, Competence Progression, and Task Scheduling Shape Training Dynamics
In standard large language model pre-training and fine-tuning pipelines, training batches are almost universally sampled uniformly and independently at random from a static corpus: $$\mathcal{D} = \{z_i = (x_i, y_i)\}_{i=1}^N$$ While this independent and identically distributed (i.i.d.) sampling paradigm aligns with empirical risk minimization (ERM), it ignores the non-convex geometry of deep transformer loss surfaces. Early in training, when network parameters are randomly initialized or unal
August 25, 20261 min