Subscribe
Resources / Tutorials / page 5

Tutorials

116 articles

Articles, page 5

81-100 of 116

  1. The Fisher Information Matrix and Natural Gradient Descent: How Information Geometry, Riemannian Manifolds, and Curvature Invariance Reshape Neural Optimization

    The Fisher Information Matrix and Natural Gradient Descent: How Information Geometry, Riemannian Manifolds, and Curvature Invariance Reshape Neural Optimization Standard gradient descent operates under an implicit, often unexamined assumption: that parameter space is Euclidean. When an optimizer updates network weights via $\theta_{t+1} = \theta_t - \eta \nabla_\theta \mathcal{L}(\theta)$, it calculates the direction of steepest descent within an arbitrary coordinate system. If the network is r

    AI
    10 min read

  2. Teacher Forcing and Exposure Bias in Autoregressive Models: Mathematical Foundations, Compounding Errors, and Mitigation Strategies

    Teacher Forcing and Exposure Bias in Autoregressive Models: Mathematical Foundations, Compounding Errors, and Mitigation Strategies Autoregressive sequence models form the computational backbone of modern language modeling, code synthesis, and multi-step reasoning agents. From decoder-only Transformers like GPT and Claude to recurrent architectures, these systems factorize the joint probability of a sequence of tokens $y = (y_1, y_2, \dots, y_T)$ into a product of conditional probabilities: $$

    AI
    10 min read

  3. Implicit Bias of Gradient Descent: How Optimization Geometry Replaces Explicit Regularization

    When engineers train a neural network with plain stochastic gradient descent and no weight decay, the result often generalizes instead of collapsing into an overfit mess. Classical learning theory predicts disaster: with more parameters than data points, unregularized training should find a memorizing interpolating solution with no predictive power. Instead, gradient descent reliably lands on solutions that look chosen. This systematic preference — the implicit bias of gradient descent — is one

    AI
    6 min read

  4. Supervised Contrastive Learning: How Multi-Positive InfoNCE and Geometric Alignment Outperform Cross-Entropy

    Supervised Contrastive Learning: How Multi-Positive InfoNCE and Geometric Alignment Outperform Cross-Entropy For decades, the categorical cross-entropy loss function served as the default objective for supervised neural network training. By minimizing the negative log-likelihood of ground-truth class logits, cross-entropy drives neural network backpropagation across computer vision, natural language processing, and speech recognition. Despite its ubiquity, cross-entropy introduces structural sh

    AI
    8 min read

  5. The Edge of Stability: How Progressive Sharpening and Hessian Dynamics Govern Deep Learning Optimization

    In classical convex optimization, the behavior of gradient descent is dictated by the Lipschitz smoothness constant of the objective function. If a function $f(\theta)$ has an $L$-smooth gradient—meaning the largest eigenvalue of its Hessian matrix is bounded by $\lambda_{\max}(\nabla^2 f(\theta)) \le L$—gradient descent with learning rate $\eta$ is guaranteed to monotonically reduce the loss if and only if $\eta < 2/L$. When the step size exceeds this threshold ($\eta > 2/\lambda_{\max}$), stan

    AI
    7 min read

  6. Centered Kernel Alignment: How CKA Measures Representation Similarity Across Layers and Architectures

    Understanding how deep neural networks represent information across layers, training steps, and disparate architectures has long been a central challenge in machine learning interpretability. When two neural networks are trained on the exact same dataset, even from identical model architectures, their learned weight matrices and individual neuron activations differ completely due to random initialization, data shuffling, and non-convex optimization. Because representations are not aligned to a s

    AI
    9 min read

  7. Kolmogorov-Arnold Networks: How Learnable Edge Splines Challenge the Multi-Layer Perceptron

    Kolmogorov-Arnold Networks: How Learnable Edge Splines Challenge the Multi-Layer Perceptron For over three decades, the foundational building block of deep neural networks has remained essentially unchanged: the Multi-Layer Perceptron (MLP). In standard transformer feed-forward blocks, convolutional networks, and deep classifiers, MLPs alternate linear matrix multiplications ($\mathbf{W}\mathbf{x}$) with fixed, point-wise non-linear activation functions (such as ReLU, GELU, or SwiGLU). Under th

    Explainers
    9 min read

  8. Sparse Attention and BigBird: How Window, Global, and Random Graphs Preserve Turing Completeness in Linear Time

    Standard self-attention in transformer architectures scales quadratically with sequence length. Computing full pairwise interactions between n tokens requires evaluating an n x n attention matrix, yielding O(n^2) computational complexity and memory consumption. While hardware accelerators and IO-aware tiling algorithms like FlashAttention optimize memory traffic, the quadratic compute and KV footprint remains a barrier for processing long contexts, document-level summarization, and genomic seque

    AI
    7 min read

  9. Forward KL vs. Reverse KL Divergence: Mode Covering, Mode Seeking, and the Alignment Dynamics of Large Language Models

    Forward KL vs. Reverse KL Divergence: Mode Covering, Mode Seeking, and the Alignment Dynamics of Large Language Models Every phase of modern large language model development, from multi-trillion token pre-training to reinforcement learning from human feedback (RLHF) and student model distillation, fundamentally revolves around minimizing statistical distance between probability distributions. The primary mathematical tool utilized for this purpose is the Kullback-Leibler (KL) divergence, introd

    AI
    9 min read

  10. The Gumbel-Softmax Trick: How Continuous Relaxations Enable Differentiable Discrete Sampling

    The Gumbel-Softmax Trick: How Continuous Relaxations Enable Differentiable Discrete Sampling In modern deep learning, end-to-end training depends on reverse-mode automatic differentiation. When an architecture operates on continuous tensors, computing gradients via the chain rule is straightforward. However, many foundational artificial intelligence problems involve discrete choices: selecting tokens from a fixed vocabulary, routing tokens to expert networks in a Mixture-of-Experts (MoE) archit

    Education
    11 min read

  11. The Information Bottleneck in Deep Learning: How Mutual Information Compression Shapes Generalization and Neural Representations

    The Information Bottleneck in Deep Learning: How Mutual Information Compression Shapes Generalization and Neural Representations Classical statistical learning theory struggles to explain why overparameterized deep neural networks generalize well to unseen test data. Traditional complexity measures such as Vapnik-Chervonenkis (VC) dimension and Rademacher complexity scale with the raw count of tunable weights, predicting severe overfitting when parameter counts exceed dataset sample sizes. Yet

    AI
    10 min read

  12. Feed-Forward Networks as Key-Value Memories: How Transformers Store, Retrieve, and Edit Parametric Knowledge

    In transformer language models, multi-head self-attention and feed-forward networks (FFNs) perform distinct computational roles. While self-attention mechanisms route contextual information dynamically across sequence positions, feed-forward layers account for roughly two-thirds of total model parameters in standard architectures. Despite early interpretations characterizing FFNs simply as generic non-linear projections, mechanistic analysis shows that feed-forward layers operate as unnormalized

    AI
    6 min read

  13. Modern Hopfield Networks: How Continuous Energy Landscapes Explain Transformer Attention and Exponential Memory

    When Vaswani et al. introduced the Transformer architecture in 2017, scaled dot-product self-attention was presented primarily as a pragmatic computational mechanism: an efficient, highly parallelizable alternative to recurrence and convolutions. By computing pairwise inner products between queries and keys, normalizing via softmax, and taking a weighted sum of values, attention allowed models to route information dynamically across arbitrarily distant tokens. For several years, self-attention

    AI
    10 min read

  14. Deep Double Descent: Why Overparameterization Defies the Classical Bias-Variance Trade-Off

    For decades, statistical learning theory rested on a foundational tenet: the bias-variance trade-off. According to classical machine learning textbooks, increasing model capacity reduces bias on the training set but inevitably inflates variance on unseen test data. The resulting risk curve forms a familiar U-shape: underfitting on the left, an optimal capacity in the center, and severe overfitting on the right. Modern deep learning and large language models (LLMs) fundamentally contradicted thi

    AI
    7 min read

  15. Minimum Bayes Risk Decoding: How Utility Consensus and Distributional Reranking Outperform Mode-Seeking in Large Language Models

    Autoregressive large language models generate text by estimating conditional probability distributions over token sequences. In conventional inference pipelines, generating the final output sequence relies almost entirely on search algorithms designed to locate high-probability trajectories: greedy decoding, beam search, or stochastic sampling with temperature and top-p filtering. However, theoretical and empirical analyses demonstrate that the most probable sequence under a model's distributio

    AI
    7 min read

  16. Reward Model Overoptimization in Large Language Models: How Goodhart's Law, Proxy Exploitation, and KL Drift Degrade Alignment

    Post-training alignment of large language models relies on optimizing a policy toward objectives defined by human intent and preferences. Because querying human evaluators during every step of continuous reinforcement learning or high-throughput rejection sampling is computationally and logistically infeasible, alignment workflows construct a parameterised proxy reward model. Trained on pairwise preference datasets through formulations such as the Bradley-Terry model, this proxy acts as a surrog

    AI
    6 min read

  17. Anthropic Launches Claude Academy and 4D AI Fluency Framework for Workforce Training

    Anthropic has launched Claude Academy, an interactive training platform aimed at standardizing how individuals and enterprise teams learn, deploy, and evaluate AI systems. Available via academy.claude.com and directly inside the Claude profile interface, the program couples tool-specific training modules with a foundational curriculum designed to cultivate systematic AI interaction patterns. The initiative comes as enterprise adoption shifts from ad-hoc prompting toward autonomous agent workflo

    Anthropic
    2 min read

  18. Token-Free and Byte-Level Language Models: How Hierarchical Patching, MegaByte, and MambaByte Eliminate Tokenizer Bottlenecks

    Modern large language models universally rely on subword tokenizers such as Byte-Pair Encoding (BPE), WordPiece, and Unigram algorithms. These tokenizers compress text into discrete integer IDs from a fixed vocabulary, typically spanning 32,000 to 256,000 entries. By collapsing three to five characters into a single token, tokenizers reduce sequence length ($L$), making quadratic $O(L^2)$ self-attention computationally tractable. However, subword tokenization introduces systemic architectural l

    AI
    7 min read

  19. Data Mixing and Domain Scheduling in Large Language Models: How DoReMi, RegMix, and Multi-Stage Annealing Shape Pre-Training Dynamics

    Data Mixing and Domain Scheduling in Large Language Models: How DoReMi, RegMix, and Multi-Stage Annealing Shape Pre-Training Dynamics In large language model pre-training, data composition is as consequential as parameter count and compute budget. While early foundation models relied on raw natural frequencies or manual heuristic filtering to construct training corpora, empirical scaling laws have shown that arbitrary domain ratios cause severe compute inefficiencies. Over-sampling redundant te

    AI
    6 min read

  20. ReAct in Large Language Models: How Interleaving Reasoning and Action Traces Built the Foundation of AI Agents

    ReAct in Large Language Models: How Interleaving Reasoning and Action Traces Built the Foundation of AI Agents Before autonomous agents could interact reliably with APIs, search engines, and bash environments, large language models (LLMs) operated in one of two disconnected paradigms: internal reasoning without external interaction, or external action generation without internal deliberation. In pure reasoning paradigms such as Chain-of-Thought (CoT) prompting, models generate intermediate natu

    AI
    10 min read