Subscribe
Topic / page 2

Deep Learning

Articles, page 2

21-40
  1. Kronecker-Factored Approximate Curvature (K-FAC) and Shampoo: How Kronecker Preconditioning Scales Second-Order Optimization

    Kronecker-Factored Approximate Curvature (K-FAC) and Shampoo: How Kronecker Preconditioning Scales Second-Order Optimization Standard deep neural network pre-training relies almost exclusively on first-order stochastic gradient methods. Optimizers such as AdamW maintain running averages of individual parameter gradients and uncentered second moments, applying a coordinate-wise diagonal preconditioner. While computationally efficient, diagonal scaling ignores off-diagonal cross-layer and intra-l

    AI
    9 min read
  2. The Edge of Stability: How Progressive Sharpening and Hessian Dynamics Govern Deep Learning Optimization

    In classical convex optimization, the behavior of gradient descent is dictated by the Lipschitz smoothness constant of the objective function. If a function $f(\theta)$ has an $L$-smooth gradient—meaning the largest eigenvalue of its Hessian matrix is bounded by $\lambda_{\max}(\nabla^2 f(\theta)) \le L$—gradient descent with learning rate $\eta$ is guaranteed to monotonically reduce the loss if and only if $\eta < 2/L$. When the step size exceeds this threshold ($\eta > 2/\lambda_{\max}$), stan

    AI
    7 min read
  3. Cross-Datacenter Distributed LLM Training in Production: DiLoCo, Local SGD, Communication Compression, and High-Latency Fault Tolerance

    Cross-Datacenter Distributed LLM Training in Production: DiLoCo, Local SGD, Communication Compression, and High-Latency Fault Tolerance Scaling frontier large language model pre-training within a single datacenter is encountering severe physical limits. Hyperscalers and AI laboratories increasingly face localized power grid saturation, where individual datacenter campuses cannot secure the 500 megawatt to multi-gigawatt utility allocations required for next-generation clusters. Consequently, in

    AI
    9 min read
  4. Relative Positional Encodings in Transformers: How Shaw's Attention, Transformer-XL, and T5 Relative Biases Preserve Translation Invariance

    Relative Positional Encodings in Transformers: How Shaw's Attention, Transformer-XL, and T5 Relative Biases Preserve Translation Invariance Standard self-attention operations in transformer architectures possess no inherent awareness of sequence order. Because scaled dot-product attention computes interactions across sets of tokens without regard to index ordering, early models relied on absolute positional encodings to inject sequential structure. While absolute encodings assign rigid coordina

    AI
    9 min read
  5. Kolmogorov-Arnold Networks: How Learnable Edge Splines Challenge the Multi-Layer Perceptron

    Kolmogorov-Arnold Networks: How Learnable Edge Splines Challenge the Multi-Layer Perceptron For over three decades, the foundational building block of deep neural networks has remained essentially unchanged: the Multi-Layer Perceptron (MLP). In standard transformer feed-forward blocks, convolutional networks, and deep classifiers, MLPs alternate linear matrix multiplications ($\mathbf{W}\mathbf{x}$) with fixed, point-wise non-linear activation functions (such as ReLU, GELU, or SwiGLU). Under th

    Explainers
    9 min read
  6. Normalizing Flows and Real NVP: How Invertible Neural Networks and Triangular Jacobians Compute Exact Log-Likelihoods

    Normalizing Flows and Real NVP: How Invertible Neural Networks and Triangular Jacobians Compute Exact Log-Likelihoods Generative modeling in deep learning revolves around a fundamental question: how can a neural network learn to transform a simple, analytically tractable probability distribution into a complex, high-dimensional empirical data distribution? Over the past decade, four primary generative modeling paradigms have emerged to address this challenge: 1. Generative Adversarial Networ

    AI
    15 min read
  7. Deep Equilibrium Models: How Implicit Layers and Root-Finding Eliminate Activation Memory in Deep Learning

    Deep Equilibrium Models: How Implicit Layers and Root-Finding Eliminate Activation Memory in Deep Learning Deep neural networks derive their expressive capacity from composition. In conventional architectures, depth is explicit: an input vector is transformed sequentially through a chain of $L$ discrete layers, each parameterized by dedicated weights. While effective, this paradigm creates a fundamental computational dependency where training memory scales linearly with network depth ($O(L)$) b

    AI
    9 min read
  8. Neural Ordinary Differential Equations: How Continuous-Depth Dynamics and Adjoint Sensitivity Solve the Memory Bottleneck in Deep Learning

    Neural Ordinary Differential Equations: How Continuous-Depth Dynamics and Adjoint Sensitivity Solve the Memory Bottleneck in Deep Learning Deep neural networks are traditionally structured as a discrete sequence of layers. An input tensor passes through layer after layer, transforming its representation at fixed, integer time steps. In standard architectures like Residual Networks (ResNets), each successive block computes an additive update: h_{t+1} = h_t + f(h_t, \theta_t) In 2018, researche

    AI
    7 min read
  9. Masked Autoencoders: How Asymmetric Encoders, High Masking Ratios, and Pixel Reconstruction Scaled Vision Transformers

    Masked Autoencoders: How Asymmetric Encoders, High Masking Ratios, and Pixel Reconstruction Scaled Vision Transformers Self-supervised pre-training transformed natural language processing through masked language modeling, popularized by BERT (Devlin et al., 2018). By hiding a subset of input tokens and training a bidirectional Transformer to predict the missing words from context, models learned rich, generalizable linguistic representations without manual annotations. Adapting this masked pre

    AI
    11 min read
  10. Gradient Low-Rank Projection (GaLore): How Dynamic Subspaces Enable Full-Parameter LLM Pre-Training with Low-Rank Memory

    Pre-training foundation large language models has historically required massive compute clusters, largely due to the memory footprint of optimizer states rather than the model weights themselves. While parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA) freeze weights and introduce small adapter matrices, they fail when applied to pre-training from scratch because they restrict parameter updates to a fixed, static low-rank manifold. Gradient Low-Rank Projection (GaLore),

    AI
    8 min read
  11. Variational Autoencoders: Mathematical Derivation of the ELBO, the Reparameterization Trick, and Mitigating Posterior Collapse

    Variational Autoencoders: Mathematical Derivation of the ELBO, the Reparameterization Trick, and Mitigating Posterior Collapse Traditional autoencoders map high-dimensional data into deterministic latent vectors. While effective for dimensionality reduction and non-linear feature compression, deterministic autoencoders fail as generative models because their latent representations lack continuous probabilistic structure. Unregularized latent spaces contain wide regions of empty space and severe

    AI
    11 min read
  12. Vector Quantization and VQ-VAEs: How Discrete Codebooks, Straight-Through Estimators, and Commitment Losses Power Multimodal Tokenization

    Autoregressive sequence models excel at discrete token prediction. In natural language processing, words and subwords map onto categorical vocabularies where token identity is exact and cross-entropy loss provides direct likelihood optimization. Continuous multi-dimensional signals—such as images, video frames, raw audio waveforms, and robotic sensorimotor trajectories—present a fundamental mismatch for standard transformer architectures. Historically, variational autoencoders (VAEs) bridged ra

    AI
    7 min read
  13. The Gumbel-Softmax Trick: How Continuous Relaxations Enable Differentiable Discrete Sampling

    The Gumbel-Softmax Trick: How Continuous Relaxations Enable Differentiable Discrete Sampling In modern deep learning, end-to-end training depends on reverse-mode automatic differentiation. When an architecture operates on continuous tensors, computing gradients via the chain rule is straightforward. However, many foundational artificial intelligence problems involve discrete choices: selecting tokens from a fixed vocabulary, routing tokens to expert networks in a Mixture-of-Experts (MoE) archit

    Education
    11 min read
  14. The Information Bottleneck in Deep Learning: How Mutual Information Compression Shapes Generalization and Neural Representations

    The Information Bottleneck in Deep Learning: How Mutual Information Compression Shapes Generalization and Neural Representations Classical statistical learning theory struggles to explain why overparameterized deep neural networks generalize well to unseen test data. Traditional complexity measures such as Vapnik-Chervonenkis (VC) dimension and Rademacher complexity scale with the raw count of tunable weights, predicting severe overfitting when parameter counts exceed dataset sample sizes. Yet

    AI
    10 min read
  15. The Lottery Ticket Hypothesis in Large Language Models: How Sparse Subnetworks and Iterative Magnitude Pruning Retain Transformer Capacity

    The Lottery Ticket Hypothesis in Large Language Models: How Sparse Subnetworks and Iterative Magnitude Pruning Retain Transformer Capacity Modern large language models operate under extreme overparameterization. Frontier architectures allocate tens or hundreds of billions of parameters to achieve low perplexity and robust generalization across reasoning, code generation, and factual retrieval. Yet empirical pruning consistently demonstrates that post-training models can lose 30% to 50% of their

    AI
    9 min read
  16. Linear Mode Connectivity in Deep Neural Networks: How Permutation Symmetries, Git Re-Basin, and the Single-Basin Hypothesis Unify Model Checkpoints

    The classical intuition for high-dimensional optimization assumes that deep neural network loss landscapes are rugged, non-convex terrains populated by isolated local minima. Under this view, two models trained independently from distinct random initializations settle into distant, incompatible basins separated by insurmountable loss barriers. Direct parameter averaging between two such models typically degrades accuracy to random chance. Research over the past several years has overturned this

    Deep Learning
    9 min read
  17. The Curse of Multilinguality in Large Language Models: Capacity Dilution, Tokenizer Fertility, and Representation Interference

    The Curse of Multilinguality in Large Language Models: Capacity Dilution, Tokenizer Fertility, and Representation Interference Training a single transformer foundation model to process dozens or hundreds of languages is one of the central goals of modern natural language processing. In theory, massive multilingual pre-training unlocks positive cross-lingual transfer: low-resource languages gain syntactic, factual, and reasoning capabilities from the rich supervision available in high-resource l

    AI
    7 min read
  18. Logit Soft-Capping in Large Language Models: How Tanh-Based Bounds Prevent Attention Saturation and Numerical Instability

    In modern autoregressive Transformer training, maintaining numerical stability across trillions of tokens requires strict control over intermediate activation magnitudes. As models grow deeper and wider, pre-softmax logits in self-attention mechanisms and final vocabulary projection layers frequently drift toward extreme values. When logit values grow unconstrained, standard softmax normalization enters a saturation regime where output probabilities collapse into near one-hot distributions, caus

    AI
    7 min read
  19. Reversible Transformers: How Invertible Residual Blocks Eliminate Activation Memory in Deep Networks

    Training deep transformer models is primarily bounded by activation memory rather than parameter storage. During the forward pass of standard backpropagation, automatic differentiation engines cache intermediate activations across every attention head, layer normalization, and feed-forward sublayer so they can be referenced during the backward pass to evaluate gradients. For a transformer with N layers, sequence length L, batch size B, and hidden dimension d_model, storing these activations requ

    AI
    6 min read
  20. Weight Tying in Large Language Models: Mathematical Foundations, Geometric Bottlenecks, and Modern Architectural Trade-Offs

    Weight Tying in Large Language Models: Mathematical Foundations, Geometric Bottlenecks, and Modern Architectural Trade-Offs In autoregressive language models, the embedding layer at the input and the unembedding projection layer at the output serve as the two bridges between discrete vocabulary tokens and the continuous hidden representation space. In the foundational Transformer architecture (Vaswani et al., 2017) and early generative models like GPT-2 (Radford et al., 2019), the weights of th

    AI
    8 min read