Subscribe
Topic

Representation Learning

How neural networks learn useful representations without labels. Posts explain contrastive methods like SimCLR, MoCo and InfoNCE, non-contrastive methods like Barlow Twins and VICReg, and the alignment and uniformity view of contrastive learning.

Articles

1-4
  1. Alignment and Uniformity on the Hypersphere: How Geometric Losses Govern Contrastive Representation Learning

    Alignment and Uniformity on the Hypersphere: The Geometric Foundations of Contrastive Representation Learning Contrastive representation learning serves as the foundational objective behind modern neural embeddings, powering dense retrieval systems, visual-language models such as CLIP, and metric learning pipelines. While early literature justified contrastive learning through the InfoMax principle (maximizing mutual information between augmented views), theoretical and empirical analyses have

    Machine Learning
    11 min read
  2. SimCLR and MoCo: How Momentum Encoders, Dynamic Queues, and InfoNCE Scaled Self-Supervised Representation Learning

    Self-supervised visual representation learning underwent a foundational transformation in 2020 with the introduction of contrastive learning frameworks. Prior to this period, pre-training computer vision models without human annotations relied on heuristic pretext tasks such as jigsaw puzzle solving, rotation prediction, or autoencoding color channels. These methods forced networks to learn low-level geometric or statistical artifacts rather than generalizable semantic abstractions. Contrastive

    AI
    8 min read
  3. Non-Contrastive Representation Learning: How Barlow Twins and VICReg Prevent Feature Collapse via Redundancy Reduction

    Self-supervised representation learning provides the foundation for modern foundation models across computer vision, audio, and multimodal systems. By training deep neural networks to produce compact vector embeddings without human annotations, self-supervised pre-training enables models to capture rich semantic structures directly from raw data. Historically, the dominant approach to self-supervised learning was contrastive learning, popularized by architectures such as SimCLR (Chen et al., 20

    edu
    9 min read
  4. Noise-Contrastive Estimation and InfoNCE: How Partition Function Estimation and Mutual Information Lower Bounds Power Modern AI

    Noise-Contrastive Estimation and InfoNCE: How Partition Function Elimination and Mutual Information Lower Bounds Power Modern AI In statistical machine learning and generative modeling, evaluating the exact probability of an observed event frequently requires calculating a normalizing constant known as the partition function. For continuous spaces or discrete spaces with high cardinality, such as a natural language vocabulary spanning over one hundred thousand tokens or high-dimensional pixel d

    AI
    10 min read