Subscribe
Resources / Tutorials / page 3

Tutorials

116 articles

Articles, page 3

41-60 of 116

  1. Titans: Learning to Memorize at Test Time — Mathematical Foundations of Neural Long-Term Memory, Surprise-Driven Gradient Updates, Adaptive Forgetting, and Context Scaling

    Sequence modeling in large language models has long been caught in a fundamental trade-off between attention mechanisms and recurrent architectures. Softmax attention provides precise associative recall over all tokens in the active context window, but incurs quadratic computational and memory complexity, $O(N^2)$, with respect to sequence length $N$. Conversely, linear recurrent models, such as State Space Models (Mamba, Mamba-2) and linear Transformers (GLA, RetNet, DeltaNet), compress sequenc

    AI
    8 min read

  2. Gated Linear Attention (GLA): Mathematical Foundations, Data-Dependent Gating, Chunkwise Parallel Forms, and Sub-Quadratic Sequence Modeling

    Standard Transformer architectures rely on softmax self-attention, which exhibits quadratic time and memory complexity $O(N^2)$ with respect to sequence length $N$. While FlashAttention and its successors optimize memory access patterns to execute exact softmax attention efficiently within GPU SRAM, the fundamental quadratic compute barrier remains. Moreover, autoregressive generation requires maintaining a Key-Value (KV) cache that grows linearly with sequence length, creating a memory capacity

    AI
    9 min read

  3. Self-Taught Reasoner (STaR) and Quiet-STaR: Mathematical Foundations, Rationalization Dynamics, and Bootstrapped Reasoning in Large Language Models

    Self-Taught Reasoner (STaR) and Quiet-STaR: Mathematical Foundations, Rationalization Dynamics, and Bootstrapped Reasoning in Large Language Models Chain-of-thought prompting enables autoregressive transformers to solve multi-step arithmetic, symbolic logic, and commonsense reasoning tasks by emitting intermediate tokens before generating a final answer. However, scaling chain-of-thought capabilities faces a fundamental data bottleneck: supervised datasets of high-quality, human-annotated reaso

    AI
    9 min read

  4. Differential Transformer (Diff Transformer): Mathematical Foundations of Differential Attention, Noise Cancellation, and Hallucination Suppression

    Differential Transformer (Diff Transformer): Mathematical Foundations of Differential Attention, Noise Cancellation, and Hallucination Suppression Standard Transformer architectures rely on softmax attention to route information between tokens in an autoregressive sequence. While effective across scale, softmax attention suffers from a structural tendency to allocate non-trivial probability mass to irrelevant context tokens. In long-context retrieval, multi-hop reasoning, and document summariza

    Education
    6 min read

  5. Diffusion Transformers (DiT): Mathematical Foundations, Patchification, Adaptive LayerNorm (adaLN-Zero), and Latent Diffusion Scaling

    Diffusion Transformers (DiT), introduced by William Peebles and Saining Xie in their 2023 paper Scalable Diffusion Models with Transformers, established a structural departure in generative visual modeling. For years, diffusion pipelines relied almost exclusively on convolutional U-Net backbones derived from Denoising Diffusion Probabilistic Models (DDPM) and Latent Diffusion Models (LDM). DiT replaced these multi-scale convolutional networks with standard isotropic Transformer architectures ope

    AI
    5 min read

  6. Sigmoid Loss for Language Image Pre-Training (SigLIP): Mathematical Foundations, Pairwise Binary Classification, Softmax Normalization Removal, and Batch Size Scaling in Multimodal Vision-Language Models

    Executive Summary Multimodal foundation models rely heavily on contrastive representation learning to map visual features and natural language tokens into a unified embedding space. For years, the dominant paradigm was the InfoNCE objective popularized by CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), which uses categorical cross-entropy with a softmax normalization over all paired elements in a mini-batch. While effective, the softmax normalization imposes a severe distributed syste

    AI
    8 min read

  7. Tree of Thoughts (ToT): Mathematical Foundations, Thought Decomposition, State Evaluation, and Tree Search Algorithms in Large Language Models

    Autoregressive large language models generate text left-to-right via token-level greedy or stochastic decoding. While Chain-of-Thought (CoT) prompting enables models to generate intermediate reasoning steps, standard inference remains constrained to a single linear trajectory. In complex problem-solving environments requiring combinatorial planning, heuristic evaluation, and lookahead, single-trajectory generation suffers from compounding errors and an inability to backtrack from early dead ends

    AI
    6 min read

  8. Stochastic Decoding Strategies in Large Language Models: Mathematical Foundations of Temperature Scaling, Top-k, Nucleus (Top-p), Typical Sampling, and Min-p

    Autoregressive language models generate text sequentially by estimating a conditional probability distribution over a vocabulary at each decoding step. While model training optimizes cross-entropy loss over large corpora, the quality, diversity, and coherence of generated sequences depend directly on the decoding strategy used to select tokens from the output logits. Deterministic algorithms such as greedy search and beam search systematically fail in open-ended generation tasks, collapsing int

    AI
    8 min read

  9. Machine Unlearning in Large Language Models: Mathematical Foundations of Gradient Ascent, Negative Preference Optimization, and Representation Misdirection

    Machine Unlearning in Large Language Models: Mathematical Foundations of Gradient Ascent, Negative Preference Optimization, and Representation Misdirection Machine unlearning in large language models (LLMs) addresses the challenge of removing specific training data, copyrighted material, private identifiers, or hazardous capabilities from a pre-trained model without retraining the network from scratch. Because training frontier language models requires millions of GPU-hours and petabyte-scale c

    AI
    8 min read

  10. The Logit Lens and Tuned Lens: Mathematical Foundations, Residual Stream Projections, and Representation Dynamics in Transformers

    Large language models based on decoder-only transformer architectures process information through a sequence of stacked attention and multi-layer perceptron (MLP) blocks connected by a residual stream. While the final output distribution over the vocabulary is produced by projecting the final hidden state through an unembedding matrix, understanding how intermediate representations evolve across depth remains a central problem in mechanistic interpretability. The logit lens and the tuned lens p

    AI
    5 min read

  11. Query-Key Normalization (QK-Norm): Mathematical Foundations, Attention Logit Drift, Entropy Collapse, and Large-Scale Transformer Training Stability

    Large-scale Transformer pre-training at trillion-token scale frequently encounters sudden, irrecoverable training instabilities, often manifesting as gradient norm spikes and divergence in training loss. While gradient clipping, learning rate warmups, and pre-layer normalization (Pre-LN) mitigate optimization bottlenecks in shallow architectures, deeper and wider models exhibit an intrinsic failure mode in the standard scaled dot-product attention mechanism: unconstrained attention logit growth.

    AI
    6 min read

  12. Representation Fine-Tuning (ReFT): Mathematical Foundations, Low-Rank Linear Subspaces, Intervention Operations, and Parameter-Efficient Representation Editing in Large Language Models

    Representation Fine-Tuning (ReFT) introduces a parameter-efficient adaptation paradigm that operates directly on the hidden representations of frozen language models rather than updating or augmenting model weight matrices. Developed by researchers at Stanford University (Wu et al., 2024), ReFT leverages insights from mechanistic interpretability and causal abstraction to steer model behavior through learned low-rank subspace interventions. By editing internal activations at targeted sequence po

    AI
    6 min read

  13. Activation Patching and Causal Tracing: Mathematical Foundations, Counterfactual Interventions, and Circuit Discovery in Large Language Models

    Activation Patching and Causal Tracing: Mathematical Foundations, Counterfactual Interventions, and Circuit Discovery in Large Language Models Mechanistic interpretability aims to reverse-engineer the computational graphs implemented by trained neural networks into human-understandable algorithms. In autoregressive Transformer models, standard observational tools such as attention map visualization and linear probing often fail to distinguish between active computational paths and passive corre

    Education
    8 min read

  14. Self-Play Fine-Tuning (SPIN): Mathematical Foundations, Two-Player Minimax Games, and Iterative Distribution Alignment

    Self-Play Fine-Tuning (SPIN): Mathematical Foundations, Two-Player Minimax Games, and Iterative Distribution Alignment Supervised Fine-Tuning (SFT) has served as the baseline method for converting pre-trained foundation models into instruction-following assistants. By training on curated prompt-response pairs $(x, y)$ via Maximum Likelihood Estimation (MLE), SFT aligns the model's next-token distributions with human demonstrations. However, standard SFT exhibits fundamental theoretical limitat

    AI
    8 min read

  15. Weight Pruning in Large Language Models: Mathematical Foundations of Magnitude Pruning, Wanda, SparseGPT, and Semi-Structured 2:4 Sparsity

    Weight Pruning in Large Language Models: Mathematical Foundations of Magnitude Pruning, Wanda, SparseGPT, and Semi-Structured 2:4 Sparsity As frontier transformer models expand past hundreds of billions of parameters, model serving is constrained by memory capacity and memory bandwidth. In autoregressive token generation, memory bandwidth dominates latency because memory-bound matrix-vector operations require streaming entire weight matrices from High Bandwidth Memory (HBM) to on-chip SRAM for

    Education
    7 min read

  16. Identity Preference Optimization (IPO): Mathematical Foundations, Overfitting in DPO, and Non-Parametric Preference Alignment

    Identity Preference Optimization (IPO) is a closed-form preference alignment algorithm for large language models developed by researchers at Google DeepMind. Introduced by Azar et al. (2023) in A General Theoretical Paradigm to Understand Learning from Human Preferences, IPO addresses a fundamental failure mode in Direct Preference Optimization (DPO): empirical and theoretical overfitting caused by unbounded surrogate losses and rigid Bradley-Terry preference assumptions. While DPO recasts rein

    AI
    8 min read

  17. Flash-Decoding: Mathematical Foundations, Split-KV Parallelism, Online Softmax Reduction, and Serving Latency Optimization

    Large language model inference consists of two distinct computational regimes: the compute-bound prefill phase and the memory-bandwidth-bound autoregressive decoding phase. While the prefill phase processes prompt tokens in parallel across dense matrix multiplication units, generation proceeds token by token. At each generation step, a single newly generated query vector must attend to the entire historical Key-Value (KV) cache stored in High-Bandwidth Memory (HBM). As context windows scale fro

    AI
    8 min read

  18. Chain-of-Thought and Computational Expressivity in Transformers: Mathematical Foundations, Circuit Complexity Limits, and Sequential Reasoning Mechanics

    Standard autoregressive Transformers generate text by computing token representations through a fixed stack of attention and feed-forward layers. When evaluated on complex reasoning, mathematical derivation, or multi-step algorithmic execution, standard models attempting to produce direct answers frequently fail. Introducing intermediate reasoning tokens through scratchpads and Chain-of-Thought (CoT) prompting eliminates these failure modes across diverse problem domains. While early empirical

    AI
    6 min read

  19. Maximal Update Parametrization (muP): Mathematical Foundations, Infinite-Width Scaling Limits, and Zero-Shot Hyperparameter Transfer in LLM Pre-Training

    Pre-training large language models requires optimizing sensitive hyperparameters, including learning rate, weight initialization variance, Adam momentum coefficients, and learning rate schedules. In standard neural network implementations, optimal hyperparameters change whenever model dimensions expand. A learning rate tuned on a 125-million parameter model frequently destabilizes or underperforms when applied directly to a 13-billion or 70-billion parameter architecture. Because grid-searching

    AI
    5 min read

  20. Constitutional AI and RLAIF: Mathematical Foundations, Critique-Revision Loops, and Scalable AI-Supervised Alignment

    Constitutional AI and RLAIF: Mathematical Foundations, Critique-Revision Loops, and Scalable AI-Supervised Alignment Standard Reinforcement Learning from Human Feedback (RLHF) has served as the baseline alignment mechanism for frontier large language models since the introduction of InstructGPT. However, human-in-the-loop preference curation introduces severe engineering and scaling constraints. Human crowdworkers are expensive, slow, susceptible to cognitive fatigue, and inconsistent when eval

    Education
    7 min read