Educational25 articles

Educational

Articles

KV Cache Eviction and Streaming Attention: Mathematical Foundations, Attention Sink Mechanics, Heavy Hitter Oracles (H2O), and Bounded-Memory Generation
KV Cache Eviction and Streaming Attention: Mathematical Foundations, Attention Sink Mechanics, Heavy Hitter Oracles (H2O), and Bounded-Memory Generation In autoregressive transformer generation, memory consumption and serving throughput are dominated by the Key-Value (KV) cache. For long-context generation and continuous multi-turn dialogue, the linear growth of the KV cache with sequence length imposes an unsustainable memory footprint and saturates high-bandwidth GPU memory channels. Standar
August 28, 20261 min
Byte-Pair Encoding (BPE) and Modern Subword Tokenization: Mathematical Foundations, Merge Dynamics, Byte-Level Encodings, and Vocabulary Compression Mechanics
Byte-Pair Encoding (BPE) and Modern Subword Tokenization: Mathematical Foundations, Merge Dynamics, Byte-Level Encodings, and Vocabulary Compression Mechanics Tokenization is the discrete boundary interface between raw textual sequences and the continuous vector representations of autoregressive large language models. Before an attention block computes a single inner product or a feed-forward network applies an activation function, an input string must be mapped into an ordered sequence of disc
August 28, 20261 min
Direct Preference Optimization (DPO): Mathematical Foundations, Implicit Reward Reparameterization, Bradley-Terry Policy Derivation, and Closed-Form Alignment Dynamics
Direct Preference Optimization (DPO): Mathematical Foundations, Implicit Reward Reparameterization, Bradley-Terry Policy Derivation, and Closed-Form Alignment Dynamics Post-training alignment is the critical bridge that transforms raw auto-regressive language models into coherent, steerable, and safe assistants. For several years, the standard approach to preference alignment relied on Reinforcement Learning from Human Feedback (RLHF) executed via Proximal Policy Optimization (PPO). While conce
August 28, 20261 min
Rotary Position Embeddings (RoPE) and Context Window Extension: Mathematical Foundations, Complex Rotations, NTK-Aware Scaling, and YaRN Dynamics
Autoregressive transformers process input sequences as permutation-invariant collections of token vectors. Without explicit positional encoding, the self-attention mechanism cannot distinguish between different token orderings. While early transformer architectures relied on additive absolute position embeddings (such as learned position tables or fixed sinusoidal encodings) or additive relative position biases, modern frontier large language models have almost universally converged on Rotary Po
August 28, 20261 min
RingAttention and Context Parallelism: Mathematical Foundations, Block-Wise Exact Attention Over Ring Topologies, Overlapped Peer-to-Peer Communication, and Infinite-Context Scaling
RingAttention and Context Parallelism: Mathematical Foundations, Block-Wise Exact Attention Over Ring Topologies, Overlapped Peer-to-Peer Communication, and Infinite-Context Scaling Scaling context windows from thousands to millions of tokens has transformed large language models from localized sequence processors into comprehensive repository analyzers, long-form document synthesizers, and multi-modal world simulators. However, standard self-attention exhibits quadratic computational and memor
August 28, 20261 min
Simple Preference Optimization (SimPO): Mathematical Foundations, Reference-Free Implicit Reward, Length Normalization, and Target Margin Dynamics
Simple Preference Optimization (SimPO): Mathematical Foundations, Reference-Free Implicit Reward, Length Normalization, and Target Margin Dynamics Post-training preference alignment has become the definitive step in transforming raw pretrained large language models into instruction-following assistants. While Reinforcement Learning from Human Feedback (RLHF) via Proximal Policy Optimization (PPO) established the initial standard, its requirement to maintain actor, critic, reference, and reward
August 28, 20261 min
Contrastive Preference Optimization (CPO): Mathematical Foundations, Direct Likelihood Regularization, Out-of-Distribution Error Bounds, and Non-Degenerate Policy Alignment
Supervised fine-tuning (SFT) and preference alignment represent the standard two-stage training paradigm for post-training large language models. While SFT establishes foundational task adherence by minimizing cross-entropy loss against target references, it suffers from two structural limitations: it cannot surpass the quality ceiling of its training demonstrations, and it lacks an explicit mechanism to penalize near-correct outputs that contain subtle hallucinations or omissions. Direct Prefe
August 28, 20261 min
Knowledge Distillation in Large Language Models: Mathematical Foundations of Forward vs. Reverse KL Divergence, Sequence-Level Distillation (SeqKD), and On-Policy Mode-Seeking Objectives
Knowledge Distillation in Large Language Models: Mathematical Foundations of Forward vs. Reverse KL Divergence, Sequence-Level Distillation (SeqKD), and On-Policy Mode-Seeking Objectives Deploying frontier large language models in latency-critical and compute-constrained environments remains an active challenge in artificial intelligence systems. While parameter scale correlates directly with downstream generalization and emergent reasoning capabilities, the memory footprint and FLOP intensity
August 27, 20261 min
State Space Models (SSMs) and Mamba: Mathematical Foundations of Continuous-Time Discretization, Selective Scan Mechanics, and Structured State Space Duality
State Space Models (SSMs) and Mamba: Mathematical Foundations of Continuous-Time Discretization, Selective Scan Mechanics, and Structured State Space Duality Autoregressive Large Language Models (LLMs) built on the standard Transformer architecture rely on softmax self-attention. While self-attention provides expressive in-context learning and dense token-to-token associative recall, it exhibits fundamental algorithmic scaling limits: prefill compute scales quadratically with sequence length ($
August 27, 20261 min
Byte-Pair Encoding (BPE) and Modern Subword Tokenization: Mathematical Foundations, Merge Dynamics, Byte-Level Encodings, and Vocabulary Compression Mechanics
Byte-Pair Encoding (BPE) and Modern Subword Tokenization: Mathematical Foundations, Merge Dynamics, Byte-Level Encodings, and Vocabulary Compression Mechanics Tokenization is the discrete boundary interface between raw textual sequences and the continuous vector representations of autoregressive large language models. Before an attention block computes a single inner product or a feed-forward network applies an activation function, an input string must be mapped into an ordered sequence of disc
August 27, 20261 min
Post-Training Quantization (PTQ): Mathematical Foundations of Optimal Brain Surgeon, GPTQ Hessian Inversion, AWQ Salient Scaling, and SmoothQuant Outlier Migration
Post-Training Quantization (PTQ): Mathematical Foundations of Optimal Brain Surgeon, GPTQ Hessian Inversion, AWQ Salient Scaling, and SmoothQuant Outlier Migration Serving modern large language models at scale requires addressing severe hardware constraints. In autoregressive generation, decoding is fundamentally bounded by memory bandwidth rather than floating-point computation throughput. Each generated token requires reading every parameter from High Bandwidth Memory (HBM) into SRAM to perfo
August 27, 20261 min
Speculative Decoding: Mathematical Foundations, Rejection Sampling Dynamics, Draft Architectures, and Serving Latency
Autoregressive language models generate text sequentially, producing one token per forward pass. Because each forward pass must load hundreds of billions of parameters from high-bandwidth memory (HBM) into compute registers to calculate the next token for a small batch, decoding operates in a memory-bandwidth-bound regime with low arithmetic intensity. Speculative decoding resolves this bottleneck. Introduced independently by Leviathan et al. (2022) and Chen et al. (2023), the technique uses a
August 26, 20261 min
Ring Attention with Blockwise Transformers: Mathematical Foundations, Circular KV Passing, Overlapped Communication-Computation, and Linear Context Scaling
Scaling the context length of transformer-based large language models has historically collided with two fundamental walls: the $O(S^2)$ computational and memory complexity of standard self-attention, and the high-bandwidth memory (HBM) capacity of individual accelerator devices. While IO-aware tiling algorithms such as FlashAttention eliminate intermediate quadratic activation storage by computing softmax within fast on-chip SRAM, the entire sequence key and value tensors must still reside with
August 25, 20261 min
Implicit Bias of Gradient Descent: How Optimization Geometry Replaces Explicit Regularization
title: "Implicit Bias of Gradient Descent: How Optimization Geometry Replaces Explicit Regularization" feature_image: "https://cms.llms.blog/content/images/2026/08/implicit-bias-cover.png" status: "published" When engineers train a neural network with plain stochastic gradient descent and no weight decay, the result often generalizes instead of collapsing into an overfit mess. Classical learning theory predicts disaster: with more parameters than data points, unregularized training should find
August 24, 20261 min
Minimum Bayes Risk Decoding: How Utility Consensus and Distributional Reranking Outperform Mode-Seeking in Large Language Models
Autoregressive large language models generate text by estimating conditional probability distributions over token sequences. In conventional inference pipelines, generating the final output sequence relies almost entirely on search algorithms designed to locate high-probability trajectories: greedy decoding, beam search, or stochastic sampling with temperature and top-p filtering. However, theoretical and empirical analyses demonstrate that the most probable sequence under a model's distributio
August 22, 20261 min
Data Mixing and Domain Scheduling in Large Language Models: How DoReMi, RegMix, and Multi-Stage Annealing Shape Pre-Training Dynamics
Data Mixing and Domain Scheduling in Large Language Models: How DoReMi, RegMix, and Multi-Stage Annealing Shape Pre-Training Dynamics In large language model pre-training, data composition is as consequential as parameter count and compute budget. While early foundation models relied on raw natural frequencies or manual heuristic filtering to construct training corpora, empirical scaling laws have shown that arbitrary domain ratios cause severe compute inefficiencies. Over-sampling redundant te
August 21, 20261 min
Self-Consistency Decoding: How Sampling Diverse Reasoning Paths Eliminates Single-Trajectory Fragility in LLMs
Self-Consistency Decoding: How Sampling Diverse Reasoning Paths Eliminates Single-Trajectory Fragility in LLMs Standard inference pipelines in large language models rely on greedy decoding or low-temperature sampling to generate answers. For tasks requiring direct factual recall or basic classification, greedy token selection generates fluent and accurate outputs. However, when large models are tasked with complex multi-step reasoning, greedy generation exhibits severe structural fragility. In
August 20, 20261 min
Rejection Sampling Fine-Tuning: How Filtering Model Outputs by Reward Replaced RLHF Complexity
Rejection Sampling Fine-Tuning: How Filtering Model Outputs by Reward Replaced RLHF Complexity Rejection sampling fine-tuning (RAFT) has emerged as the practical workhorse of LLM alignment. While PPO-based RLHF dominated early literature, production systems from Llama 2 to DeepSeek-R1 rely on a simpler loop: generate multiple completions per prompt, score them with a reward model, keep the best, and fine-tune on the filtered data. The technique converts the reinforcement learning problem into s
August 20, 20261 min
SwiGLU and Gated Linear Units: How Bilinear Gating Replaced Standard FFNs in Modern LLMs
Every modern open-weight and frontier large language model, from Meta's LLaMA 3 and Mistral to Alibaba's Qwen 2.5 and DeepSeek-V3, has abandoned the standard two-layer Feed-Forward Network (FFN) originally introduced in the 2017 Transformer architecture. In its place, model architectures have converged almost universally on Gated Linear Units (GLU), specifically the Swish-Gated Linear Unit (SwiGLU). While the original Transformer relied on standard non-linear activations like ReLU or Gaussian E
August 19, 20261 min
Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck
Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck Autoregressive large language models built on standard multi-head self-attention face two fundamental scaling ceilings: quadratic compute and memory complexity during pre-training, and linearly expanding key-value (KV) cache memory footprints during autoregressive generation. While optimizations such as FlashAttention reduce memory access overheads and Grouped-Query Attention (GQ
August 19, 20261 min