Subscribe
Topic / page 12

Explainers

Articles, page 12

221-240
  1. Transfusion: Mathematical Foundations of Unified Discrete Next-Token Prediction and Continuous Latent Diffusion in Multimodal Transformers

    Generative foundation models have historically operated under two distinct mathematical paradigms: autoregressive sequence modeling over discrete vocabularies (used in text and code generation) and iterative denoising diffusion over continuous vector spaces (used in image and video synthesis). Early attempts to unify both modalities in a single end-to-end transformer forced continuous perceptual data into discrete codebooks using vector quantization (such as VQ-VAE or VQGAN). While discretizatio

    Education
    6 min read
  2. Self-Taught Reasoner (STaR) and Quiet-STaR: Mathematical Foundations, Rationalization Dynamics, and Bootstrapped Reasoning in Large Language Models

    Self-Taught Reasoner (STaR) and Quiet-STaR: Mathematical Foundations, Rationalization Dynamics, and Bootstrapped Reasoning in Large Language Models Chain-of-thought prompting enables autoregressive transformers to solve multi-step arithmetic, symbolic logic, and commonsense reasoning tasks by emitting intermediate tokens before generating a final answer. However, scaling chain-of-thought capabilities faces a fundamental data bottleneck: supervised datasets of high-quality, human-annotated reaso

    AI
    9 min read
  3. Differential Transformer (Diff Transformer): Mathematical Foundations of Differential Attention, Noise Cancellation, and Hallucination Suppression

    Differential Transformer (Diff Transformer): Mathematical Foundations of Differential Attention, Noise Cancellation, and Hallucination Suppression Standard Transformer architectures rely on softmax attention to route information between tokens in an autoregressive sequence. While effective across scale, softmax attention suffers from a structural tendency to allocate non-trivial probability mass to irrelevant context tokens. In long-context retrieval, multi-hop reasoning, and document summariza

    Education
    6 min read
  4. Diffusion Transformers (DiT): Mathematical Foundations, Patchification, Adaptive LayerNorm (adaLN-Zero), and Latent Diffusion Scaling

    Diffusion Transformers (DiT), introduced by William Peebles and Saining Xie in their 2023 paper Scalable Diffusion Models with Transformers, established a structural departure in generative visual modeling. For years, diffusion pipelines relied almost exclusively on convolutional U-Net backbones derived from Denoising Diffusion Probabilistic Models (DDPM) and Latent Diffusion Models (LDM). DiT replaced these multi-scale convolutional networks with standard isotropic Transformer architectures ope

    AI
    5 min read
  5. Sigmoid Loss for Language Image Pre-Training (SigLIP): Mathematical Foundations, Pairwise Binary Classification, Softmax Normalization Removal, and Batch Size Scaling in Multimodal Vision-Language Models

    Executive Summary Multimodal foundation models rely heavily on contrastive representation learning to map visual features and natural language tokens into a unified embedding space. For years, the dominant paradigm was the InfoNCE objective popularized by CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), which uses categorical cross-entropy with a softmax normalization over all paired elements in a mini-batch. While effective, the softmax normalization imposes a severe distributed syste

    AI
    8 min read
  6. Tree of Thoughts (ToT): Mathematical Foundations, Thought Decomposition, State Evaluation, and Tree Search Algorithms in Large Language Models

    Autoregressive large language models generate text left-to-right via token-level greedy or stochastic decoding. While Chain-of-Thought (CoT) prompting enables models to generate intermediate reasoning steps, standard inference remains constrained to a single linear trajectory. In complex problem-solving environments requiring combinatorial planning, heuristic evaluation, and lookahead, single-trajectory generation suffers from compounding errors and an inability to backtrack from early dead ends

    AI
    6 min read
  7. Stochastic Decoding Strategies in Large Language Models: Mathematical Foundations of Temperature Scaling, Top-k, Nucleus (Top-p), Typical Sampling, and Min-p

    Autoregressive language models generate text sequentially by estimating a conditional probability distribution over a vocabulary at each decoding step. While model training optimizes cross-entropy loss over large corpora, the quality, diversity, and coherence of generated sequences depend directly on the decoding strategy used to select tokens from the output logits. Deterministic algorithms such as greedy search and beam search systematically fail in open-ended generation tasks, collapsing int

    AI
    8 min read
  8. Representation Fine-Tuning (ReFT): Mathematical Foundations, Low-Rank Linear Subspaces, Intervention Operations, and Parameter-Efficient Representation Editing in Large Language Models

    Representation Fine-Tuning (ReFT) introduces a parameter-efficient adaptation paradigm that operates directly on the hidden representations of frozen language models rather than updating or augmenting model weight matrices. Developed by researchers at Stanford University (Wu et al., 2024), ReFT leverages insights from mechanistic interpretability and causal abstraction to steer model behavior through learned low-rank subspace interventions. By editing internal activations at targeted sequence po

    AI
    6 min read
  9. Self-Play Fine-Tuning (SPIN): Mathematical Foundations, Two-Player Minimax Games, and Iterative Distribution Alignment

    Self-Play Fine-Tuning (SPIN): Mathematical Foundations, Two-Player Minimax Games, and Iterative Distribution Alignment Supervised Fine-Tuning (SFT) has served as the baseline method for converting pre-trained foundation models into instruction-following assistants. By training on curated prompt-response pairs $(x, y)$ via Maximum Likelihood Estimation (MLE), SFT aligns the model's next-token distributions with human demonstrations. However, standard SFT exhibits fundamental theoretical limitat

    AI
    10 min read
  10. Identity Preference Optimization (IPO): Mathematical Foundations, Overfitting in DPO, and Non-Parametric Preference Alignment

    Identity Preference Optimization (IPO) is a closed-form preference alignment algorithm for large language models developed by researchers at Google DeepMind. Introduced by Azar et al. (2023) in A General Theoretical Paradigm to Understand Learning from Human Preferences, IPO addresses a fundamental failure mode in Direct Preference Optimization (DPO): empirical and theoretical overfitting caused by unbounded surrogate losses and rigid Bradley-Terry preference assumptions. While DPO recasts rein

    AI
    8 min read
  11. Flash-Decoding: Mathematical Foundations, Split-KV Parallelism, Online Softmax Reduction, and Serving Latency Optimization

    Large language model inference consists of two distinct computational regimes: the compute-bound prefill phase and the memory-bandwidth-bound autoregressive decoding phase. While the prefill phase processes prompt tokens in parallel across dense matrix multiplication units, generation proceeds token by token. At each generation step, a single newly generated query vector must attend to the entire historical Key-Value (KV) cache stored in High-Bandwidth Memory (HBM). As context windows scale fro

    AI
    8 min read
  12. Maximal Update Parametrization (muP): Mathematical Foundations, Infinite-Width Scaling Limits, and Zero-Shot Hyperparameter Transfer in LLM Pre-Training

    Pre-training large language models requires optimizing sensitive hyperparameters, including learning rate, weight initialization variance, Adam momentum coefficients, and learning rate schedules. In standard neural network implementations, optimal hyperparameters change whenever model dimensions expand. A learning rate tuned on a 125-million parameter model frequently destabilizes or underperforms when applied directly to a 13-billion or 70-billion parameter architecture. Because grid-searching

    AI
    5 min read
  13. Constitutional AI and RLAIF: Mathematical Foundations, Critique-Revision Loops, and Scalable AI-Supervised Alignment

    Constitutional AI and RLAIF: Mathematical Foundations, Critique-Revision Loops, and Scalable AI-Supervised Alignment Standard Reinforcement Learning from Human Feedback (RLHF) has served as the baseline alignment mechanism for frontier large language models since the introduction of InstructGPT. However, human-in-the-loop preference curation introduces severe engineering and scaling constraints. Human crowdworkers are expensive, slow, susceptible to cognitive fatigue, and inconsistent when eval

    Education
    7 min read
  14. Attention with Linear Biases (ALiBi): Mathematical Foundations, Static Positional Biases, and Context Extrapolation in Transformers

    Standard Transformer architectures rely on positional encodings to inject sequence order into permutation-invariant self-attention operations. Early models utilized absolute sinusoidal encodings or learned position embeddings, while subsequent iterations introduced relative positional embeddings and Rotary Position Embedding (RoPE). However, models trained with standard absolute or rotary encodings historically struggled with length extrapolation: evaluating sequences longer than the pre-trainin

    AI
    6 min read
  15. Kahneman-Tversky Optimization (KTO): Mathematical Foundations, Prospect Theory, and Binary Signal Alignment in Large Language Models

    Post-training alignment of large language models has long relied on pairwise comparison datasets. Methods such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) assume access to curated pairs $(x, y_w, y_l)$, where a human or automated judge explicitly marks completion $y_w$ as superior to $y_l$ for a given prompt $x$. In production environments, however, collecting paired preferences is logistically complex, expensive, and artificial. Real-world telem

    Explainers
    10 min read
  16. Curriculum Learning in Large Language Models: How Difficulty Pacing, Competence Progression, and Task Scheduling Shape Training Dynamics

    In standard large language model pre-training and fine-tuning pipelines, training batches are almost universally sampled uniformly and independently at random from a static corpus: $$\mathcal{D} = \{z_i = (x_i, y_i)\}_{i=1}^N$$ While this independent and identically distributed (i.i.d.) sampling paradigm aligns with empirical risk minimization (ERM), it ignores the non-convex geometry of deep transformer loss surfaces. Early in training, when network parameters are randomly initialized or unal

    Explainers
    12 min read
  17. The Edge of Stability: How Progressive Sharpening and Hessian Dynamics Govern Deep Learning Optimization

    In classical convex optimization, the behavior of gradient descent is dictated by the Lipschitz smoothness constant of the objective function. If a function $f(\theta)$ has an $L$-smooth gradient—meaning the largest eigenvalue of its Hessian matrix is bounded by $\lambda_{\max}(\nabla^2 f(\theta)) \le L$—gradient descent with learning rate $\eta$ is guaranteed to monotonically reduce the loss if and only if $\eta < 2/L$. When the step size exceeds this threshold ($\eta > 2/\lambda_{\max}$), stan

    AI
    7 min read
  18. Kolmogorov-Arnold Networks: How Learnable Edge Splines Challenge the Multi-Layer Perceptron

    Kolmogorov-Arnold Networks: How Learnable Edge Splines Challenge the Multi-Layer Perceptron For over three decades, the foundational building block of deep neural networks has remained essentially unchanged: the Multi-Layer Perceptron (MLP). In standard transformer feed-forward blocks, convolutional networks, and deep classifiers, MLPs alternate linear matrix multiplications ($\mathbf{W}\mathbf{x}$) with fixed, point-wise non-linear activation functions (such as ReLU, GELU, or SwiGLU). Under th

    Explainers
    9 min read