theory3 articles

theory

Articles

The Information Bottleneck in Deep Learning: How Mutual Information Compression Shapes Generalization and Neural Representations
The Information Bottleneck in Deep Learning: How Mutual Information Compression Shapes Generalization and Neural Representations Classical statistical learning theory struggles to explain why overparameterized deep neural networks generalize well to unseen test data. Traditional complexity measures such as Vapnik-Chervonenkis (VC) dimension and Rademacher complexity scale with the raw count of tunable weights, predicting severe overfitting when parameter counts exceed dataset sample sizes. Yet
August 23, 20261 min
Neural Collapse: How Simplex Equiangular Tight Frames Emerge at the Terminal Phase of Training
In classification tasks, deep neural networks exhibit an unexpected geometric simplicity during late-stage optimization. While the internal activations of early training appear high-dimensional and complex, the penultimate layer representations and linear classifiers converge toward an exact, symmetrical geometric structure known as Neural Collapse (NC). First identified empirically by Papyan, Han, and Donoho (2020), Neural Collapse emerges during the Terminal Phase of Training (TPT). This regi
August 22, 20261 min
Neural Tangent Kernel: How Infinite-Width Networks Linearize Gradient Descent
title: "Neural Tangent Kernel: How Infinite-Width Networks Linearize Gradient Descent" slug: "neural-tangent-kernel-how-infinite-width-networks-linearize-gradient-descent" feature_image: "https://cms.llms.blog/content/images/2026/08/neural-tangent-kernel-cover.png" tags: ["edu", "theory", "foundations"] status: published The Neural Tangent Kernel (NTK) describes the behavior of infinitely wide neural networks during gradient descent. In the infinite-width limit, network training reduces to kern
August 22, 20261 min

theory

Articles

The Information Bottleneck in Deep Learning: How Mutual Information Compression Shapes Generalization and Neural Representations

Neural Collapse: How Simplex Equiangular Tight Frames Emerge at the Terminal Phase of Training

Neural Tangent Kernel: How Infinite-Width Networks Linearize Gradient Descent