Subscribe
Topic / page 3

Machine Learning

Articles, page 3

41-60
  1. Temperature Scaling and Model Calibration in Deep Neural Networks: Mathematical Foundations, Expected Calibration Error, and Post-Hoc Logit Optimization

    Modern deep neural networks achieve high classification accuracy and generative benchmark performance across vision, language, and decision-making tasks. However, optimization for raw accuracy does not ensure that predicted softmax probabilities correspond to true posterior probabilities. A model that assigns a 0.90 probability to an output should be correct exactly 90% of the time. When empirical accuracy systematically diverges from predicted confidence, the model is miscalibrated. Research b

    Deep Learning
    11 min read
  2. Loss Landscapes in Deep Neural Networks: How Filter Normalization, Hessian Curvature, and Basin Flatness Explain Generalization

    The parameter space of modern deep learning models spans millions to hundreds of billions of dimensions. In this high-dimensional space, the empirical risk objective forms a complex geometric surface known as the loss landscape. Despite the extreme non-convexity of deep neural networks, standard first-order optimization algorithms such as stochastic gradient descent (SGD) and Adam regularly converge to parameter configurations that achieve both low training error and robust test set generalizati

    AI
    9 min read
  3. Alignment and Uniformity on the Hypersphere: How Geometric Losses Govern Contrastive Representation Learning

    Alignment and Uniformity on the Hypersphere: The Geometric Foundations of Contrastive Representation Learning Contrastive representation learning serves as the foundational objective behind modern neural embeddings, powering dense retrieval systems, visual-language models such as CLIP, and metric learning pipelines. While early literature justified contrastive learning through the InfoMax principle (maximizing mutual information between augmented views), theoretical and empirical analyses have

    Machine Learning
    11 min read
  4. Teacher Forcing and Exposure Bias in Autoregressive Models: Mathematical Foundations, Compounding Errors, and Mitigation Strategies

    Teacher Forcing and Exposure Bias in Autoregressive Models: Mathematical Foundations, Compounding Errors, and Mitigation Strategies Autoregressive sequence models form the computational backbone of modern language modeling, code synthesis, and multi-step reasoning agents. From decoder-only Transformers like GPT and Claude to recurrent architectures, these systems factorize the joint probability of a sequence of tokens $y = (y_1, y_2, \dots, y_T)$ into a product of conditional probabilities: $$

    AI
    10 min read
  5. Integrated Gradients: How Axiomatic Attribution Solves the Gradients-at-Saturation Problem in Deep Neural Networks

    Integrated Gradients: How Axiomatic Attribution Solves the Gradients-at-Saturation Problem in Deep Neural Networks Feature attribution methods in deep learning aim to answer a fundamental interpretability question: given an input vector and a trained neural network, how much did each input dimension contribute to the model's final output score? In natural language processing and computer vision, practitioners routinely need to identify which input tokens, pixels, or tabular variables drove a sp

    AI
    10 min read
  6. Stochastic Weight Averaging: How Geometric Ensembling Finds Flatter Optima and Improves Generalization

    Stochastic Weight Averaging (SWA): How Geometric Ensembling Finds Flatter Optima and Improves Generalization During the optimization of deep neural networks, standard Stochastic Gradient Descent (SGD) and adaptive optimizers often struggle to find solutions that generalize robustly to unseen data. While learning rate decay allows optimizers to settle into local minima on the empirical training loss surface, empirical and theoretical analyses reveal that standard gradient descent tends to halt n

    AI
    8 min read
  7. Sequence Packing and Loss Masking in Production SFT: Architecture, Trade-Offs, and Framework Implementations

    Sequence Packing and Loss Masking in Production SFT: Architecture, Trade-Offs, and Framework Implementations Supervised fine-tuning pipelines routinely waste 40–70% of FLOPs on padding tokens. Sequence packing eliminates that waste by concatenating multiple training examples into a single long tensor, but it introduces three hard constraints: attention must not leak across example boundaries, loss must be computed only on the intended target spans (typically assistant responses), and the data c

    AI
    6 min read
  8. Data Pruning and Core-Set Selection in Deep Learning: How EL2N, GraNd, and Memorization Dynamics Break Power-Law Scaling

    Modern foundation models are trained on tens of trillions of tokens, requiring millions of GPU hours. Yet empirical analyses consistently reveal that massive portions of web-crawled corpora and large-scale vision datasets are either highly redundant, uninformative, or dominated by unlearnable noise. Standard empirical scaling laws (such as those formulated by Kaplan et al. and Chinchilla) model generalization error as a power-law function of total training samples ($L(N) \propto N^{-\alpha}$). H

    AI
    7 min read
  9. Energy-Based Models: How Energy Landscapes, Contrastive Divergence, and Langevin Dynamics Unify Generative Learning

    Energy-Based Models: How Energy Landscapes, Contrastive Divergence, and Langevin Dynamics Unify Generative Learning Probabilistic modeling in machine learning fundamentally centers on estimating data distributions over high-dimensional spaces. Standard generative architectures achieve this by enforcing structural constraints: autoregressive models factorize joint distributions through causal chains, normalizing flows constrain network architectures to invertible bijections with tractable Jacobi

    AI
    9 min read
  10. Implicit Bias of Gradient Descent: How Optimization Geometry Replaces Explicit Regularization

    When engineers train a neural network with plain stochastic gradient descent and no weight decay, the result often generalizes instead of collapsing into an overfit mess. Classical learning theory predicts disaster: with more parameters than data points, unregularized training should find a memorizing interpolating solution with no predictive power. Instead, gradient descent reliably lands on solutions that look chosen. This systematic preference — the implicit bias of gradient descent — is one

    AI
    6 min read
  11. Supervised Contrastive Learning: How Multi-Positive InfoNCE and Geometric Alignment Outperform Cross-Entropy

    Supervised Contrastive Learning: How Multi-Positive InfoNCE and Geometric Alignment Outperform Cross-Entropy For decades, the categorical cross-entropy loss function served as the default objective for supervised neural network training. By minimizing the negative log-likelihood of ground-truth class logits, cross-entropy drives neural network backpropagation across computer vision, natural language processing, and speech recognition. Despite its ubiquity, cross-entropy introduces structural sh

    AI
    8 min read
  12. SimCLR and MoCo: How Momentum Encoders, Dynamic Queues, and InfoNCE Scaled Self-Supervised Representation Learning

    Self-supervised visual representation learning underwent a foundational transformation in 2020 with the introduction of contrastive learning frameworks. Prior to this period, pre-training computer vision models without human annotations relied on heuristic pretext tasks such as jigsaw puzzle solving, rotation prediction, or autoencoding color channels. These methods forced networks to learn low-level geometric or statistical artifacts rather than generalizable semantic abstractions. Contrastive

    AI
    8 min read
  13. LLM Evaluation Arenas in Production: Bradley-Terry Modeling, Active Matchmaking, Style Bias Control, and Bootstrapped Elo Calibration

    LLM Evaluation Arenas in Production: Bradley-Terry Modeling, Active Matchmaking, Style Bias Control, and Bootstrapped Elo Calibration Static benchmarks such as MMLU, GSM8K, and HumanEval face severe limitations in production machine learning environments. Modern foundation models rapidly saturate static multiple-choice questions, training datasets frequently suffer from benchmark contamination, and synthetic test suites fail to capture open-ended, multi-turn user intent. Consequently, engineeri

    AI
    9 min read
  14. Centered Kernel Alignment: How CKA Measures Representation Similarity Across Layers and Architectures

    Understanding how deep neural networks represent information across layers, training steps, and disparate architectures has long been a central challenge in machine learning interpretability. When two neural networks are trained on the exact same dataset, even from identical model architectures, their learned weight matrices and individual neuron activations differ completely due to random initialization, data shuffling, and non-convex optimization. Because representations are not aligned to a s

    AI
    9 min read
  15. Kolmogorov-Arnold Networks: How Learnable Edge Splines Challenge the Multi-Layer Perceptron

    Kolmogorov-Arnold Networks: How Learnable Edge Splines Challenge the Multi-Layer Perceptron For over three decades, the foundational building block of deep neural networks has remained essentially unchanged: the Multi-Layer Perceptron (MLP). In standard transformer feed-forward blocks, convolutional networks, and deep classifiers, MLPs alternate linear matrix multiplications ($\mathbf{W}\mathbf{x}$) with fixed, point-wise non-linear activation functions (such as ReLU, GELU, or SwiGLU). Under th

    Explainers
    9 min read
  16. The Self-Taught Reasoner (STaR): How Iterative Rationale Bootstrapping and Rationalization Scale LLM Problem Solving

    Autoregressive language models often fail when forced to predict answers to multi-step reasoning problems in a single forward pass. Intermediate chain-of-thought generation allows models to allocate additional compute to difficult reasoning steps before emitting a final prediction. However, obtaining high-quality intermediate reasoning traces has traditionally presented a difficult trade-off: either rely on small prompt-based few-shot demonstrations that underperform fine-tuned models, or constr

    AI
    4 min read
  17. Deep Equilibrium Models: How Implicit Layers and Root-Finding Eliminate Activation Memory in Deep Learning

    Deep Equilibrium Models: How Implicit Layers and Root-Finding Eliminate Activation Memory in Deep Learning Deep neural networks derive their expressive capacity from composition. In conventional architectures, depth is explicit: an input vector is transformed sequentially through a chain of $L$ discrete layers, each parameterized by dedicated weights. While effective, this paradigm creates a fundamental computational dependency where training memory scales linearly with network depth ($O(L)$) b

    AI
    9 min read
  18. Denoising Diffusion Probabilistic Models and DDIM: Mathematical Foundations, Variance Schedules, and Fast Deterministic Sampling

    Generative modeling in continuous state spaces underwent a fundamental transformation with the formulation of Denoising Diffusion Probabilistic Models (DDPM) by Ho et al. (2020), building on the non-equilibrium thermodynamics foundations established by Sohl-Dickstein et al. (2015). Prior to diffusion models, deep generative synthesis was dominated by Generative Adversarial Networks (GANs), which suffered from training instability and mode collapse, and Variational Autoencoders (VAEs), which ofte

    AI
    11 min read
  19. Variational Autoencoders: Mathematical Derivation of the ELBO, the Reparameterization Trick, and Mitigating Posterior Collapse

    Variational Autoencoders: Mathematical Derivation of the ELBO, the Reparameterization Trick, and Mitigating Posterior Collapse Traditional autoencoders map high-dimensional data into deterministic latent vectors. While effective for dimensionality reduction and non-linear feature compression, deterministic autoencoders fail as generative models because their latent representations lack continuous probabilistic structure. Unregularized latent spaces contain wide regions of empty space and severe

    AI
    11 min read
  20. Non-Contrastive Representation Learning: How Barlow Twins and VICReg Prevent Feature Collapse via Redundancy Reduction

    Self-supervised representation learning provides the foundation for modern foundation models across computer vision, audio, and multimodal systems. By training deep neural networks to produce compact vector embeddings without human annotations, self-supervised pre-training enables models to capture rich semantic structures directly from raw data. Historically, the dominant approach to self-supervised learning was contrastive learning, popularized by architectures such as SimCLR (Chen et al., 20

    edu
    9 min read