Optimization31 articles

Optimization

Articles

Speculative Decoding and Speculative Sampling: Mathematical Foundations, Lossless Rejection Sampling, Draft Model Architectures, and Serving Economics
Large language models generate text autoregressively, predicting one token at a time by passing the full sequence through dozens of transformer layers. During inference at low batch sizes, this generation process is fundamentally memory-bandwidth bound rather than compute bound. Speculative decoding addresses this bottleneck by using a fast drafting mechanism to propose candidate token sequences and verifying them in parallel with the primary target model in a single forward pass. When configur
August 28, 20261 min
Reference-Free Preference Optimization: Mathematical Foundations of SimPO and ORPO, Length-Normalized Implicit Rewards, Target Margins, and Monolithic Alignment
Reference-Free Preference Optimization: Mathematical Foundations of SimPO and ORPO, Length-Normalized Implicit Rewards, Target Margins, and Monolithic Alignment Post-training alignment has shifted from complex multi-stage reinforcement learning pipelines toward direct preference optimization paradigms. While Proximal Policy Optimization (PPO) requires maintaining four concurrent models in memory (policy, value, reference, and reward networks), Direct Preference Optimization (DPO) reduced this f
August 28, 20261 min
FlashAttention: IO-Aware Tiling, Online Softmax Mathematics, Memory Hierarchy Dynamics, and Kernel Evolution
Standard multi-head attention is the computational core of transformer language models. While mathematically straightforward, standard attention implementations suffer from severe memory and throughput bottlenecks as sequence lengths grow. The bottleneck is not raw arithmetic compute, but memory traffic: repeatedly reading and writing intermediate attention matrices across GPU memory tiers. FlashAttention, introduced by Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré in 202
August 26, 20261 min
FlashAttention: Mathematical Foundations, Online Softmax Tiling, IO-Awareness, and Exact Attention Scaling
Standard multi-head self-attention in the Transformer architecture exhibits quadratic time and memory complexity with respect to sequence length $N$. While the $O(N^2)$ computational complexity is widely cited, the primary performance bottleneck in production hardware is not arithmetic throughput, but memory access overhead. On modern GPU architectures such as NVIDIA A100 and H100, tensor processing cores execute matrix multiplications at teraflop and petaflop scales, but memory bandwidth betwee
August 26, 20261 min
Transformers as Support Vector Machines: Mathematical Foundations of Implicit Bias, Margin Maximization, and Hard-Attention Convergence
Understanding why overparameterized deep neural networks generalize well despite possessing sufficient capacity to memorize training data remains a fundamental question in theoretical machine learning. In classical linear models trained on separable data under exponential-tailed losses (such as logistic or cross-entropy loss), gradient descent exhibits an implicit bias: parameter iterates diverge in norm while their directions asymptotically converge to the maximum-margin hyperplane, known as th
August 26, 20261 min
Projecting Conflicting Gradients (PCGrad): Mathematical Foundations, Orthogonal Projections, and Multi-Task Optimization in Deep Learning
Projecting Conflicting Gradients (PCGrad): Mathematical Foundations, Orthogonal Projections, and Multi-Task Optimization in Deep Learning In modern machine learning systems, models rarely optimize for a single objective. Foundation models are trained simultaneously on diverse data distributions spanning natural language, source code, mathematical reasoning, and multimodal inputs. Similarly, post-training alignment pipelines must simultaneously optimize for helpfulness, factual accuracy, harmles
August 26, 20261 min
Simple Preference Optimization (SimPO): Mathematical Foundations, Length-Normalized Implicit Rewards, and Reference-Free Margin Alignment
Aligning large language models with human preferences has evolved rapidly from complex multi-stage reinforcement learning pipelines toward direct, offline optimization objectives. While Direct Preference Optimization (DPO) eliminated the requirement for training a separate reward model and executing online policy rollouts, it introduced architectural and operational challenges of its own: the necessity of maintaining an active reference policy in GPU memory, an unnormalized log-probability formu
August 25, 20261 min
The Muon Optimizer: Mathematical Foundations, Newton-Schulz Polar Iterations, and Spectral Steepest Descent in LLM Pre-Training
Transformer pre-training has long relied on AdamW as the default first-order optimizer. While AdamW provides coordinate-wise scale invariance through running estimates of first and second gradient moments, it treats every weight parameter as an isolated scalar. For modern neural networks composed primarily of two-dimensional linear projections, attention transformations, and feed-forward weight matrices, this coordinate-wise factorization neglects the structural transformation properties of matr
August 25, 20261 min
Decoupled Weight Decay Regularization (AdamW): Mathematical Foundations, L2 Regularization Failure in Adaptive Gradient Methods, and Hyperparameter Dynamics
Decoupled weight decay regularization, formalized by Ilya Loshchilov and Frank Hutter in 2017 and published at ICLR 2019, resolved a fundamental flaw in how adaptive gradient algorithms like Adam implemented weight regularization. Prior to AdamW, deep learning frameworks implemented weight decay as L2 regularization by adding the gradient of the squared L2 norm directly to the loss gradient. While mathematically identical for standard stochastic gradient descent (SGD), this formulation breaks do
August 25, 20261 min
Group Relative Policy Optimization: Mathematical Foundations, Group-Relative Baselines, and Critic-Free Reinforcement Learning in Reasoning Models
Group Relative Policy Optimization (GRPO): Mathematical Foundations, Group-Relative Baselines, and Critic-Free Reinforcement Learning in Reasoning Models Reinforcement learning from human and automated feedback has become the central mechanism for eliciting complex reasoning, tool use, and mathematical problem-solving in large language models. For years, Proximal Policy Optimization (PPO) served as the default algorithmic framework for on-policy alignment. However, standard PPO introduces signi
August 25, 20261 min
Direct Preference Optimization: Mathematical Derivation, Implicit Reward Formulation, and the Mechanics of RL-Free Alignment
Direct Preference Optimization: Mathematical Derivation, Implicit Reward Formulation, and the Mechanics of RL-Free Alignment Aligning autoregressive large language models with human preferences has traditionally relied on Reinforcement Learning from Human Feedback (RLHF). In the standard formulation popularized by InstructGPT and related post-training regimes, alignment requires a multi-stage pipeline: supervised fine-tuning (SFT), training a separate reward model on pairwise comparison data, a
August 25, 20261 min
Curriculum Learning in Large Language Models: How Difficulty Pacing, Competence Progression, and Task Scheduling Shape Training Dynamics
In standard large language model pre-training and fine-tuning pipelines, training batches are almost universally sampled uniformly and independently at random from a static corpus: $$\mathcal{D} = \{z_i = (x_i, y_i)\}_{i=1}^N$$ While this independent and identically distributed (i.i.d.) sampling paradigm aligns with empirical risk minimization (ERM), it ignores the non-convex geometry of deep transformer loss surfaces. Early in training, when network parameters are randomly initialized or unal
August 25, 20261 min
Trust Region Policy Optimization: Mathematical Foundations, Monotonic Improvement Guarantees, and Conjugate Gradient Updates
Trust Region Policy Optimization: Mathematical Foundations, Monotonic Improvement Guarantees, and Conjugate Gradient Updates In policy gradient reinforcement learning, optimization dynamics differ fundamentally from standard supervised learning. In supervised regression or classification, the underlying data distribution $P(x, y)$ remains stationary throughout training; a sub-optimal parameter update merely yields high loss on the current batch without corrupting future sample collection. In re
August 24, 20261 min
The Fisher Information Matrix and Natural Gradient Descent: How Information Geometry, Riemannian Manifolds, and Curvature Invariance Reshape Neural Optimization
The Fisher Information Matrix and Natural Gradient Descent: How Information Geometry, Riemannian Manifolds, and Curvature Invariance Reshape Neural Optimization Standard gradient descent operates under an implicit, often unexamined assumption: that parameter space is Euclidean. When an optimizer updates network weights via $\theta_{t+1} = \theta_t - \eta \nabla_\theta \mathcal{L}(\theta)$, it calculates the direction of steepest descent within an arbitrary coordinate system. If the network is r
August 24, 20261 min
Temperature Scaling and Model Calibration in Deep Neural Networks: Mathematical Foundations, Expected Calibration Error, and Post-Hoc Logit Optimization
Modern deep neural networks achieve high classification accuracy and generative benchmark performance across vision, language, and decision-making tasks. However, optimization for raw accuracy does not ensure that predicted softmax probabilities correspond to true posterior probabilities. A model that assigns a 0.90 probability to an output should be correct exactly 90% of the time. When empirical accuracy systematically diverges from predicted confidence, the model is miscalibrated. Research b
August 24, 20261 min
Loss Landscapes in Deep Neural Networks: How Filter Normalization, Hessian Curvature, and Basin Flatness Explain Generalization
The parameter space of modern deep learning models spans millions to hundreds of billions of dimensions. In this high-dimensional space, the empirical risk objective forms a complex geometric surface known as the loss landscape. Despite the extreme non-convexity of deep neural networks, standard first-order optimization algorithms such as stochastic gradient descent (SGD) and Adam regularly converge to parameter configurations that achieve both low training error and robust test set generalizati
August 24, 20261 min
The Score Function Estimator: Mathematical Foundations of REINFORCE, Log-Derivative Tricks, and Baseline Variance Reduction
In modern artificial intelligence, standard backpropagation relies on continuous differentiability: every operation between model parameters and the final loss must provide well-behaved analytical Jacobian matrices. However, many of the most critical optimization challenges in machine learning break this continuity. Autoregressive token generation in large language models, discrete tool invocation, programmatic compiler execution, and black-box reward environments are fundamentally non-different
August 24, 20261 min
Stochastic Weight Averaging: How Geometric Ensembling Finds Flatter Optima and Improves Generalization
Stochastic Weight Averaging (SWA): How Geometric Ensembling Finds Flatter Optima and Improves Generalization During the optimization of deep neural networks, standard Stochastic Gradient Descent (SGD) and adaptive optimizers often struggle to find solutions that generalize robustly to unseen data. While learning rate decay allows optimizers to settle into local minima on the empirical training loss surface, empirical and theoretical analyses reveal that standard gradient descent tends to halt n
August 24, 20261 min
Data Pruning and Core-Set Selection in Deep Learning: How EL2N, GraNd, and Memorization Dynamics Break Power-Law Scaling
Modern foundation models are trained on tens of trillions of tokens, requiring millions of GPU hours. Yet empirical analyses consistently reveal that massive portions of web-crawled corpora and large-scale vision datasets are either highly redundant, uninformative, or dominated by unlearnable noise. Standard empirical scaling laws (such as those formulated by Kaplan et al. and Chinchilla) model generalization error as a power-law function of total training samples ($L(N) \propto N^{-\alpha}$). H
August 24, 20261 min
Kronecker-Factored Approximate Curvature (K-FAC) and Shampoo: How Kronecker Preconditioning Scales Second-Order Optimization
Kronecker-Factored Approximate Curvature (K-FAC) and Shampoo: How Kronecker Preconditioning Scales Second-Order Optimization Standard deep neural network pre-training relies almost exclusively on first-order stochastic gradient methods. Optimizers such as AdamW maintain running averages of individual parameter gradients and uncentered second moments, applying a coordinate-wise diagonal preconditioner. While computationally efficient, diagonal scaling ignores off-diagonal cross-layer and intra-l
August 24, 20261 min