Subscribe
Topic / page 5

Education

Articles, page 5

81-100
  1. Weight Pruning in Large Language Models: Mathematical Foundations of Magnitude Pruning, Wanda, SparseGPT, and Semi-Structured 2:4 Sparsity

    Weight Pruning in Large Language Models: Mathematical Foundations of Magnitude Pruning, Wanda, SparseGPT, and Semi-Structured 2:4 Sparsity As frontier transformer models expand past hundreds of billions of parameters, model serving is constrained by memory capacity and memory bandwidth. In autoregressive token generation, memory bandwidth dominates latency because memory-bound matrix-vector operations require streaming entire weight matrices from High Bandwidth Memory (HBM) to on-chip SRAM for

    Education
    8 min read
  2. Identity Preference Optimization (IPO): Mathematical Foundations, Overfitting in DPO, and Non-Parametric Preference Alignment

    Identity Preference Optimization (IPO) is a closed-form preference alignment algorithm for large language models developed by researchers at Google DeepMind. Introduced by Azar et al. (2023) in A General Theoretical Paradigm to Understand Learning from Human Preferences, IPO addresses a fundamental failure mode in Direct Preference Optimization (DPO): empirical and theoretical overfitting caused by unbounded surrogate losses and rigid Bradley-Terry preference assumptions. While DPO recasts rein

    AI
    8 min read
  3. Flash-Decoding: Mathematical Foundations, Split-KV Parallelism, Online Softmax Reduction, and Serving Latency Optimization

    Large language model inference consists of two distinct computational regimes: the compute-bound prefill phase and the memory-bandwidth-bound autoregressive decoding phase. While the prefill phase processes prompt tokens in parallel across dense matrix multiplication units, generation proceeds token by token. At each generation step, a single newly generated query vector must attend to the entire historical Key-Value (KV) cache stored in High-Bandwidth Memory (HBM). As context windows scale fro

    AI
    8 min read
  4. Maximal Update Parametrization (muP): Mathematical Foundations, Infinite-Width Scaling Limits, and Zero-Shot Hyperparameter Transfer in LLM Pre-Training

    Pre-training large language models requires optimizing sensitive hyperparameters, including learning rate, weight initialization variance, Adam momentum coefficients, and learning rate schedules. In standard neural network implementations, optimal hyperparameters change whenever model dimensions expand. A learning rate tuned on a 125-million parameter model frequently destabilizes or underperforms when applied directly to a 13-billion or 70-billion parameter architecture. Because grid-searching

    AI
    5 min read
  5. Constitutional AI and RLAIF: Mathematical Foundations, Critique-Revision Loops, and Scalable AI-Supervised Alignment

    Constitutional AI and RLAIF: Mathematical Foundations, Critique-Revision Loops, and Scalable AI-Supervised Alignment Standard Reinforcement Learning from Human Feedback (RLHF) has served as the baseline alignment mechanism for frontier large language models since the introduction of InstructGPT. However, human-in-the-loop preference curation introduces severe engineering and scaling constraints. Human crowdworkers are expensive, slow, susceptible to cognitive fatigue, and inconsistent when eval

    Education
    7 min read
  6. Attention with Linear Biases (ALiBi): Mathematical Foundations, Static Positional Biases, and Context Extrapolation in Transformers

    Standard Transformer architectures rely on positional encodings to inject sequence order into permutation-invariant self-attention operations. Early models utilized absolute sinusoidal encodings or learned position embeddings, while subsequent iterations introduced relative positional embeddings and Rotary Position Embedding (RoPE). However, models trained with standard absolute or rotary encodings historically struggled with length extrapolation: evaluating sequences longer than the pre-trainin

    AI
    6 min read
  7. KV Cache Eviction and Streaming Attention: Mathematical Foundations, Attention Sink Mechanics, Heavy Hitter Oracles (H2O), and Bounded-Memory Generation

    KV Cache Eviction and Streaming Attention: Mathematical Foundations, Attention Sink Mechanics, Heavy Hitter Oracles (H2O), and Bounded-Memory Generation In autoregressive transformer generation, memory consumption and serving throughput are dominated by the Key-Value (KV) cache. For long-context generation and continuous multi-turn dialogue, the linear growth of the KV cache with sequence length imposes an unsustainable memory footprint and saturates high-bandwidth GPU memory channels. Standar

    AI
    9 min read
  8. Rotary Position Embeddings (RoPE) and Context Window Extension: Mathematical Foundations, Complex Rotations, NTK-Aware Scaling, and YaRN Dynamics

    Autoregressive transformers process input sequences as permutation-invariant collections of token vectors. Without explicit positional encoding, the self-attention mechanism cannot distinguish between different token orderings. While early transformer architectures relied on additive absolute position embeddings (such as learned position tables or fixed sinusoidal encodings) or additive relative position biases, modern frontier large language models have almost universally converged on Rotary Po

    AI
    11 min read
  9. RingAttention and Context Parallelism: Mathematical Foundations, Block-Wise Exact Attention Over Ring Topologies, Overlapped Peer-to-Peer Communication, and Infinite-Context Scaling

    Scaling context windows from thousands to millions of tokens has transformed large language models from localized sequence processors into comprehensive repository analyzers, long-form document synthesizers, and multi-modal world simulators. However, standard self-attention exhibits quadratic computational and memory complexity with respect to sequence length $S$. While IO-aware memory optimizations like FlashAttention eliminate intermediate attention matrix materialization in High Bandwidth Mem

    AI
    13 min read
  10. Anthropic Rolls Out Claude for Teachers to K-12 Schools and Districts with Free Enterprise Tier and FERPA Compliance

    Anthropic has expanded its Claude for Teachers program from individual educator access to a centralized, district-wide Enterprise offering for K-12 public and private schools. Qualifying educational organizations that enroll before June 30, 2027 receive one year of free enterprise access across their instructional and administrative staff. The update introduces centralized administrative controls, domain capture, single sign-on (SSO), role-based access control, and dedicated K-12 terms aligned

    AI
    2 min read
  11. Simple Preference Optimization (SimPO): Mathematical Foundations, Reference-Free Implicit Reward, Length Normalization, and Target Margin Dynamics

    Simple Preference Optimization (SimPO): Mathematical Foundations, Reference-Free Implicit Reward, Length Normalization, and Target Margin Dynamics Post-training preference alignment has become the definitive step in transforming raw pretrained large language models into instruction-following assistants. While Reinforcement Learning from Human Feedback (RLHF) via Proximal Policy Optimization (PPO) established the initial standard, its requirement to maintain actor, critic, reference, and reward

    AI
    11 min read
  12. OpenAI Partners with Thai Government to Launch Accelerator for Healthcare and Education Startups

    OpenAI has partnered with Thailand's Ministry of Higher Education, Science, Research and Innovation (MHESI) to launch an eight-week accelerator program supporting ten Thai startups building AI products in healthcare, wellness, and education. The initiative, announced in Bangkok, represents OpenAI's first public-private partnership focused on startup acceleration with the Thai government. Participating organizations include Thailand's National Innovation Agency (NIA), Mahidol University, and Tec

    OpenAI
    2 min read
  13. Contrastive Preference Optimization (CPO): Mathematical Foundations, Direct Likelihood Regularization, Out-of-Distribution Error Bounds, and Non-Degenerate Policy Alignment

    Supervised fine-tuning (SFT) and preference alignment represent the standard two-stage training paradigm for post-training large language models. While SFT establishes foundational task adherence by minimizing cross-entropy loss against target references, it suffers from two structural limitations: it cannot surpass the quality ceiling of its training demonstrations, and it lacks an explicit mechanism to penalize near-correct outputs that contain subtle hallucinations or omissions. Direct Prefe

    AI
    11 min read
  14. Chinchilla Scaling Laws: Mathematical Foundations of Compute-Optimal Pre-Training, IsoFLOP Loss Profiles, Parametric Power Laws, and Data-Compute Allocation

    Chinchilla Scaling Laws: Mathematical Foundations of Compute-Optimal Pre-Training, IsoFLOP Loss Profiles, Parametric Power Laws, and Data-Compute Allocation When allocating a fixed computational budget to train an autoregressive Transformer, engineers face a fundamental trade-off: should FLOPs be spent increasing the model parameter count ($N$), or should they be spent streaming a larger volume of training tokens ($D$)? For several years, frontier AI development followed the empirical scaling

    AI
    14 min read
  15. Matryoshka Representation Learning (MRL): Mathematical Foundations, Multi-Scale Loss Optimization, and Adaptive Vector Retrieval

    Matryoshka Representation Learning (MRL) has become the standard architectural foundation for modern dense text embeddings. Introduced by Kusupati et al. at NeurIPS 2022 and subsequently deployed across frontier embedding models like OpenAI text-embedding-3, Nomic Embed, and BAAI BGE-M3, MRL solves a structural inefficiency in vector retrieval: the rigid coupling between embedding dimensionality, memory consumption, and semantic fidelity. Traditional dense encoders project arbitrary text sequen

    AI
    9 min read
  16. Knowledge Distillation in Large Language Models: Mathematical Foundations of Forward vs. Reverse KL Divergence, Sequence-Level Distillation (SeqKD), and On-Policy Mode-Seeking Objectives

    Deploying frontier large language models in latency-critical and compute-constrained environments remains an active challenge in artificial intelligence systems. While parameter scale correlates directly with downstream generalization and emergent reasoning capabilities, the memory footprint and FLOP intensity of dense multi-hundred-billion parameter architectures impose prohibitive operational costs. Post-training compression strategies such as quantization and structural pruning reduce precisi

    AI
    11 min read
  17. Model Merging in Weight Space: Mathematical Foundations of Task Vectors, TIES-Merging, DARE, and Spherical Linear Interpolation

    Training large language models across multiple downstream domains typically requires expensive multi-task supervised fine-tuning (SFT) or sequential continual learning. Both paths present steep practical challenges: multi-task training demands simultaneous access to massive, curated datasets and massive compute budgets, while sequential fine-tuning suffers from catastrophic forgetting. Model merging has emerged as a data-free, compute-efficient alternative. By combining the parameters of multip

    AI
    8 min read
  18. State Space Models (SSMs) and Mamba: Mathematical Foundations of Continuous-Time Discretization, Selective Scan Mechanics, and Structured State Space Duality

    Autoregressive Large Language Models (LLMs) built on the standard Transformer architecture rely on softmax self-attention. While self-attention provides expressive in-context learning and dense token-to-token associative recall, it exhibits fundamental algorithmic scaling limits: prefill compute scales quadratically with sequence length ($O(L^2 d)$), and autoregressive token generation requires maintaining a Key-Value (KV) cache that grows linearly with sequence length ($O(L d)$). At context win

    AI
    12 min read
  19. Byte-Pair Encoding (BPE) and Modern Subword Tokenization: Mathematical Foundations, Merge Dynamics, Byte-Level Encodings, and Vocabulary Compression Mechanics

    Tokenization is the discrete boundary interface between raw textual sequences and the continuous vector representations of autoregressive large language models. Before an attention block computes a single inner product or a feed-forward network applies an activation function, an input string must be mapped into an ordered sequence of discrete integer identifiers drawn from a finite vocabulary $V$. The design and mathematical properties of this mapping govern key operational characteristics of l

    AI
    11 min read
  20. Post-Training Quantization (PTQ): Mathematical Foundations of Optimal Brain Surgeon, GPTQ Hessian Inversion, AWQ Salient Scaling, and SmoothQuant Outlier Migration

    Post-Training Quantization (PTQ): Mathematical Foundations of Optimal Brain Surgeon, GPTQ Hessian Inversion, AWQ Salient Scaling, and SmoothQuant Outlier Migration Serving modern large language models at scale requires addressing severe hardware constraints. In autoregressive generation, decoding is fundamentally bounded by memory bandwidth rather than floating-point computation throughput. Each generated token requires reading every parameter from High Bandwidth Memory (HBM) into SRAM to perfo

    AI
    10 min read