Subscribe
Topic / page 6

Education

Articles, page 6

101-120
  1. Label Smoothing Regularization: Mathematical Foundations, Maximum Entropy Dynamics, and the Regular Simplex Geometry in LLMs

    In supervised classification and autoregressive language modeling, standard cross-entropy loss trains neural networks against discrete, one-hot target distributions. While intuitive, one-hot supervision creates an inherent optimization pathology: minimizing the cross-entropy objective to zero requires pushing the logit of the target class toward positive infinity while driving all competing non-target logits toward negative infinity. In deep networks, this dynamic inflates parameter norms, cause

    AI
    11 min read
  2. SmoothQuant: Mathematical Foundations, Per-Channel Outlier Migration, and Hardware-Efficient W8A8 Inference in Large Language Models

    SmoothQuant: Mathematical Foundations, Per-Channel Outlier Migration, and Hardware-Efficient W8A8 Inference in Large Language Models Serving large language models (LLMs) in production environments presents two distinct hardware bottlenecks. During the autoregressive generation (decode) phase with small batch sizes, inference is memory-bandwidth bound, as billions of parameters must be streamed from High Bandwidth Memory (HBM) to on-chip SRAM for every generated token. Conversely, during the pro

    Quantization
    12 min read
  3. The Fisher Information Matrix and Natural Gradient Descent: How Information Geometry, Riemannian Manifolds, and Curvature Invariance Reshape Neural Optimization

    The Fisher Information Matrix and Natural Gradient Descent: How Information Geometry, Riemannian Manifolds, and Curvature Invariance Reshape Neural Optimization Standard gradient descent operates under an implicit, often unexamined assumption: that parameter space is Euclidean. When an optimizer updates network weights via $\theta_{t+1} = \theta_t - \eta \nabla_\theta \mathcal{L}(\theta)$, it calculates the direction of steepest descent within an arbitrary coordinate system. If the network is r

    AI
    10 min read
  4. Teacher Forcing and Exposure Bias in Autoregressive Models: Mathematical Foundations, Compounding Errors, and Mitigation Strategies

    Teacher Forcing and Exposure Bias in Autoregressive Models: Mathematical Foundations, Compounding Errors, and Mitigation Strategies Autoregressive sequence models form the computational backbone of modern language modeling, code synthesis, and multi-step reasoning agents. From decoder-only Transformers like GPT and Claude to recurrent architectures, these systems factorize the joint probability of a sequence of tokens $y = (y_1, y_2, \dots, y_T)$ into a product of conditional probabilities: $$

    AI
    10 min read
  5. Implicit Bias of Gradient Descent: How Optimization Geometry Replaces Explicit Regularization

    When engineers train a neural network with plain stochastic gradient descent and no weight decay, the result often generalizes instead of collapsing into an overfit mess. Classical learning theory predicts disaster: with more parameters than data points, unregularized training should find a memorizing interpolating solution with no predictive power. Instead, gradient descent reliably lands on solutions that look chosen. This systematic preference — the implicit bias of gradient descent — is one

    AI
    6 min read
  6. Supervised Contrastive Learning: How Multi-Positive InfoNCE and Geometric Alignment Outperform Cross-Entropy

    Supervised Contrastive Learning: How Multi-Positive InfoNCE and Geometric Alignment Outperform Cross-Entropy For decades, the categorical cross-entropy loss function served as the default objective for supervised neural network training. By minimizing the negative log-likelihood of ground-truth class logits, cross-entropy drives neural network backpropagation across computer vision, natural language processing, and speech recognition. Despite its ubiquity, cross-entropy introduces structural sh

    AI
    8 min read
  7. Centered Kernel Alignment: How CKA Measures Representation Similarity Across Layers and Architectures

    Understanding how deep neural networks represent information across layers, training steps, and disparate architectures has long been a central challenge in machine learning interpretability. When two neural networks are trained on the exact same dataset, even from identical model architectures, their learned weight matrices and individual neuron activations differ completely due to random initialization, data shuffling, and non-convex optimization. Because representations are not aligned to a s

    AI
    9 min read
  8. The Gumbel-Softmax Trick: How Continuous Relaxations Enable Differentiable Discrete Sampling

    The Gumbel-Softmax Trick: How Continuous Relaxations Enable Differentiable Discrete Sampling In modern deep learning, end-to-end training depends on reverse-mode automatic differentiation. When an architecture operates on continuous tensors, computing gradients via the chain rule is straightforward. However, many foundational artificial intelligence problems involve discrete choices: selecting tokens from a fixed vocabulary, routing tokens to expert networks in a Mixture-of-Experts (MoE) archit

    Education
    11 min read
  9. Deep Double Descent: Why Overparameterization Defies the Classical Bias-Variance Trade-Off

    For decades, statistical learning theory rested on a foundational tenet: the bias-variance trade-off. According to classical machine learning textbooks, increasing model capacity reduces bias on the training set but inevitably inflates variance on unseen test data. The resulting risk curve forms a familiar U-shape: underfitting on the left, an optimal capacity in the center, and severe overfitting on the right. Modern deep learning and large language models (LLMs) fundamentally contradicted thi

    AI
    7 min read
  10. Minimum Bayes Risk Decoding: How Utility Consensus and Distributional Reranking Outperform Mode-Seeking in Large Language Models

    Autoregressive large language models generate text by estimating conditional probability distributions over token sequences. In conventional inference pipelines, generating the final output sequence relies almost entirely on search algorithms designed to locate high-probability trajectories: greedy decoding, beam search, or stochastic sampling with temperature and top-p filtering. However, theoretical and empirical analyses demonstrate that the most probable sequence under a model's distributio

    AI
    7 min read
  11. Anthropic Launches Claude Academy and 4D AI Fluency Framework for Workforce Training

    Anthropic has launched Claude Academy, an interactive training platform aimed at standardizing how individuals and enterprise teams learn, deploy, and evaluate AI systems. Available via academy.claude.com and directly inside the Claude profile interface, the program couples tool-specific training modules with a foundational curriculum designed to cultivate systematic AI interaction patterns. The initiative comes as enterprise adoption shifts from ad-hoc prompting toward autonomous agent workflo

    Anthropic
    2 min read
  12. Token-Free and Byte-Level Language Models: How Hierarchical Patching, MegaByte, and MambaByte Eliminate Tokenizer Bottlenecks

    Modern large language models universally rely on subword tokenizers such as Byte-Pair Encoding (BPE), WordPiece, and Unigram algorithms. These tokenizers compress text into discrete integer IDs from a fixed vocabulary, typically spanning 32,000 to 256,000 entries. By collapsing three to five characters into a single token, tokenizers reduce sequence length ($L$), making quadratic $O(L^2)$ self-attention computationally tractable. However, subword tokenization introduces systemic architectural l

    AI
    7 min read
  13. Data Mixing and Domain Scheduling in Large Language Models: How DoReMi, RegMix, and Multi-Stage Annealing Shape Pre-Training Dynamics

    Data Mixing and Domain Scheduling in Large Language Models: How DoReMi, RegMix, and Multi-Stage Annealing Shape Pre-Training Dynamics In large language model pre-training, data composition is as consequential as parameter count and compute budget. While early foundation models relied on raw natural frequencies or manual heuristic filtering to construct training corpora, empirical scaling laws have shown that arbitrary domain ratios cause severe compute inefficiencies. Over-sampling redundant te

    AI
    6 min read
  14. Self-Consistency Decoding: How Sampling Diverse Reasoning Paths Eliminates Single-Trajectory Fragility in LLMs

    Self-Consistency Decoding: How Sampling Diverse Reasoning Paths Eliminates Single-Trajectory Fragility in LLMs Standard inference pipelines in large language models rely on greedy decoding or low-temperature sampling to generate answers. For tasks requiring direct factual recall or basic classification, greedy token selection generates fluent and accurate outputs. However, when large models are tasked with complex multi-step reasoning, greedy generation exhibits severe structural fragility. In

    AI
    6 min read
  15. SwiGLU and Gated Linear Units: How Bilinear Gating Replaced Standard FFNs in Modern LLMs

    Every modern open-weight and frontier large language model, from Meta's LLaMA 3 and Mistral to Alibaba's Qwen 2.5 and DeepSeek-V3, has abandoned the standard two-layer Feed-Forward Network (FFN) originally introduced in the 2017 Transformer architecture. In its place, model architectures have converged almost universally on Gated Linear Units (GLU), specifically the Swish-Gated Linear Unit (SwiGLU). While the original Transformer relied on standard non-linear activations like ReLU or Gaussian E

    AI
    6 min read
  16. Chinchilla Scaling Laws: How Compute-Optimal Pre-Training and Inference Economics Reshaped LLM Architecture

    Before 2022, frontier artificial intelligence labs scaled large language models primarily by increasing parameter counts while keeping training dataset sizes relatively static. OpenAI's 2020 GPT-3 model trained 175 billion parameters on 300 billion tokens. DeepMind's 2021 Gopher trained 280 billion parameters on 300 billion tokens, and Megatron-Turing NLG scaled to 530 billion parameters on 270 billion tokens. These allocation decisions followed early empirical scaling laws that suggested parame

    Education
    6 min read
  17. Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck

    Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck Autoregressive large language models built on standard multi-head self-attention face two fundamental scaling ceilings: quadratic compute and memory complexity during pre-training, and linearly expanding key-value (KV) cache memory footprints during autoregressive generation. While optimizations such as FlashAttention reduce memory access overheads and Grouped-Query Attention (GQ

    Architecture
    8 min read
  18. Context Window Extension in Large Language Models: How Position Interpolation, YaRN, and LongRoPE Scale Sequence Lengths

    Large language models are bounded during pretraining by a fixed sequence length, typically between 2,048 and 8,192 tokens. When standard autoregressive transformers attempt to process sequences beyond this pretraining context window, performance degrades immediately. Perplexity rises sharply and the model loses coherence within a few dozen tokens past the training boundary. Extending this context window by training from scratch on long sequences is computationally prohibitive due to the quadrat

    AI
    6 min read
  19. Attention Sinks in Large Language Models: How StreamingLLM Prevents Perplexity Explosion in Infinite Sequences

    Autoregressive large language models are trained on fixed context windows, yet real-world applications (such as continuous coding agents, live conversation servers, and document streaming pipelines) require models to process unbounded token sequences. When standard LLMs operate on sequences longer than their pre-training context length, computational complexity and key-value (KV) cache memory scale quadratically and linearly, respectively. A seemingly natural workaround is sliding window attent

    AI
    5 min read