Subscribe
Topic

edu

Short courses in deep learning theory and representation learning. Posts explain the Neural Tangent Kernel and how infinite-width networks linearize training, neural collapse at the end of training, and non-contrastive methods like Barlow Twins and VICReg.

Articles

1-3
  1. Non-Contrastive Representation Learning: How Barlow Twins and VICReg Prevent Feature Collapse via Redundancy Reduction

    Self-supervised representation learning provides the foundation for modern foundation models across computer vision, audio, and multimodal systems. By training deep neural networks to produce compact vector embeddings without human annotations, self-supervised pre-training enables models to capture rich semantic structures directly from raw data. Historically, the dominant approach to self-supervised learning was contrastive learning, popularized by architectures such as SimCLR (Chen et al., 20

    edu
    9 min read
  2. Neural Collapse: How Simplex Equiangular Tight Frames Emerge at the Terminal Phase of Training

    In classification tasks, deep neural networks exhibit an unexpected geometric simplicity during late-stage optimization. While the internal activations of early training appear high-dimensional and complex, the penultimate layer representations and linear classifiers converge toward an exact, symmetrical geometric structure known as Neural Collapse (NC). First identified empirically by Papyan, Han, and Donoho (2020), Neural Collapse emerges during the Terminal Phase of Training (TPT). This regi

    edu
    6 min read
  3. Neural Tangent Kernel: How Infinite-Width Networks Linearize Gradient Descent

    The Neural Tangent Kernel (NTK) describes the behavior of infinitely wide neural networks during gradient descent. In the infinite-width limit, network training reduces to kernel regression with a fixed kernel — the NTK — that stays constant throughout training. This theoretical framework explains why wide networks converge to global minima, why they struggle to learn high-frequency functions early (spectral bias), and why they behave differently from finite-width networks that learn features.

    edu
    4 min read