Subscribe
educational
7 min read

What Is Self-Supervised Learning? How AI Learned to Label Itself

The training idea behind the entire modern AI boom: hide part of the input, predict it from the rest, and let the data generate the answers instead of paying humans for them. From Shannon's 1951 guessing game to word2vec, BERT, SimCLR, CLIP and DeepSeek-R1's rule-based rewards.

Olivetti-style mid-century brochure cover reading SELF SUPERVISED LEARNING beside a grid of ochre and teal geometric shapes
Olivetti-style mid-century brochure cover reading SELF SUPERVISED LEARNING beside a grid of ochre and teal geometric shapes

Self-supervised learning is the training idea behind the entire modern AI boom. It says that a model does not need humans to tell it the right answer. It can make the data generate the answers itself: hide part of the input, ask the model to predict the hidden part from the rest, and score it against what was actually there. Text, images, speech, video and sensor logs all contain enough structure that this game produces useful understanding as a side effect.

This is not a rewrite of "unsupervised learning," and the distinction matters. Classical unsupervised learning clusters or compresses data without any target. Self-supervised learning does have targets. They are just derived from the data rather than paid for. Yann LeCun made the hierarchy famous at NIPS 2016: if intelligence is a cake, the bulk is unsupervised learning, the icing is supervised learning, and the cherry on top is reinforcement learning. In his later talks the base layer had been renamed, more precisely, to self-supervised learning.

The problem it solves: labels are the bottleneck

Deep learning's first decade ran on human-annotated data. ImageNet, the dataset that kicked off the modern era in 2009, eventually covered more than 14 million images sorted into over 20,000 categories. Building it meant filtering roughly 160 million candidate images with about 49,000 crowd workers across 167 countries on Amazon Mechanical Turk. That scale was already extraordinary in 2009. It is also structurally impossible to repeat at the size frontier models now need. A human checking each example caps the dataset at what a budget can buy.

Self-supervision removes that cap. The supervision signal costs nothing extra because it is already inside the data. Every sentence implies the word that comes next. Every image implies the pixels behind the part you cut out. Every audio clip implies the phonemes that follow. The model's homework writes itself.

The old roots of the trick

Predicting hidden parts of language is older than the phrase. In 1951 Claude Shannon measured the entropy of English by turning it into a guessing game: show a reader a text, ask them to predict the next letter given the preceding ones, and count how often they are right. From those predictions he derived that English is roughly half redundancy, which is exactly the slack self-supervised models later exploit. When BERT arrived in 2018, it described its own training task as a Cloze task, the fill-in-the-blank format from a 1953 readability study.

Masking part of the input and predicting it from the rest: the core self-supervised trick

Text figured it out first

The modern line starts with word2vec (2013). Mikolov and colleagues trained a tiny network at Google to predict a word from its surrounding context (or the context from the word) on a corpus of 1.6 billion running words from Google News. No human labeled anything. The result was dense word vectors in which king minus man plus woman got close to queen, and the training run took under a day.

Then came the sequence that produced today's chatbots:

  • GPT-1 (2018) showed that a decoder trained to predict the next token on unlabeled books and articles could be fine-tuned on a handful of labeled tasks.
  • OpenAI's GPT-2 paper (2019) made the claim blunt in its title, "Language Models are Unsupervised Multitask Learners," after training on WebText, about 8 million web pages scraped without a single annotation.
  • BERT (2018) went the other direction. It chose 15% of token positions at random; of those, 80% became a [MASK] token, 10% a random word, 10% stayed unchanged, and the model was graded on all 15%. Because the encoder could not tell which tokens it would be asked about, it had to keep a contextual representation of everything, which is why BERT beat previous models on question answering and sentiment benchmarks after light fine-tuning.

Next-token prediction, the objective every mainstream large language model still trains on, is self-supervision in its purest form: the label for position N is simply the token at position N+1 in the scraped text.

Vision took the detour through contrast

The same masking trick failed on images for years. Images are spatially smooth; predicting a hidden patch from its eight neighbors teaches a model to blur. The progress came from contrastive methods instead. Contrastive Predictive Coding (2018, DeepMind) reframed prediction as discrimination: encode the context, then pick the true future signal out of a set of decoys. The InfoNCE loss from that paper is the ancestor of most embedding models shipped today.

Contrastive learning pulls matched views together and pushes mismatched ones apart

SimCLR (2020, Google) stripped contrastive vision down to augments and a projection head. Two random crops and color jitter of the same photo formed a positive pair; everything else in a large batch formed negatives. A linear classifier trained on SimCLR's frozen features hit 76.5% top-1 on ImageNet, matching a fully supervised ResNet-50, and with 1% of the labels it outperformed AlexNet while using 100 times fewer labels. The recipe also showed contrastive learning, unlike supervised learning, keeps improving with much larger batches and longer schedules.

Masking came back and won anyway. Masked Autoencoders (2021, Facebook AI) hid 75% of an image's patches, ran a large encoder on only the visible quarter, and used a small decoder to reconstruct raw pixels. At that hiding rate interpolation stops working and the model has to represent objects and layout. Training got 3x faster because the encoder saw so few tokens, and a plain ViT-Huge reached 87.8% on ImageNet, the best result at the time among methods using only that dataset, with no labels in pre-training at all.

Pairing modalities is self-supervision too

The internet contains billions of accidental label pairs: an image and the text people chose to describe it. CLIP (2021, OpenAI) scraped 400 million image-caption pairs and trained two encoders to match which caption goes with which image, a contrastive task on naturally paired data. The zero-shot model matched the accuracy of the original supervised ResNet-50 on ImageNet without touching its 1.28 million training labels, and transferred across more than 30 vision datasets.

Speech followed. wav2vec 2.0 (2020, Facebook AI) masked the raw waveform's latent representations and solved a contrastive task over quantized speech units. Pre-trained on tens of thousands of unlabeled hours, it fine-tuned on just 10 minutes of transcribed speech (48 recordings averaging 12.5 seconds) and reached 4.8/8.2 word error rate on LibriSpeech test-clean/other. At one hour of labels it beat the previous state of the art while using 100 times less labeled data.

A standard survey from the period, Liu et al. (2020), sorted all of this into three objective families: generative (reconstruct the hidden part), contrastive (distinguish the true signal from decoys), and hybrids including adversarial setups. Nearly every frontier model since is an industrial deployment of one or more of the three.

Why the cake metaphor stuck

Self-supervision won for economic reasons before scientific ones. Labeled data costs money, time and expertise, and it runs out. Unlabeled data costs storage, and it does not run out. A model trained to predict masked tokens over billions of pages has, by construction, seen more of human knowledge than any task-specific dataset can contain, which is why one self-supervised pre-run could replace dozens of supervised ones.

The irony of the last five years is what happened on top of that base. The icing is still made by hand. Instruction tuning and RLHF put human demonstrations and preference judgments back into the pipeline: InstructGPT (2022) fine-tuned GPT-3 on about 13,000 demonstrations written by trained labelers. The bulk stayed self-supervised, but the parts users actually experience were supervised, then preference-ranked, by people.

The frontier is now closing the loop again in a new place: reasoning. DeepSeek-R1 (2025) trained models with reinforcement learning where the reward comes from rules, not raters. Math answers check against a solution; code passes or fails its tests. The paper's framing is that this "obviat[es] the need for human-labeled reasoning trajectories," letting behaviors like self-verification and strategy revision emerge in training. That is the self-supervision playbook, moved from pre-training data to reasoning data. It has a floor, though: verifiable rewards exist for math and code and almost nowhere else, so values and judgment still arrive by human supervision, or not at all.

What it means in practice

For anyone using or shipping LLMs, three consequences are worth keeping straight:

  1. Pre-training and self-supervised learning are now the same thing in all practical contexts. When a lab says it trained on 15 trillion tokens, it means next-token prediction over scraped data, no annotation involved.
  2. Fine-tuning is cheap because self-supervision did the expensive half. A specialist adapter works because the base model already knows the language, the world and the format.
  3. The supply fights are about the raw material. Web scraping, pay-per-crawl deals, robots.txt defaults and licensing lawsuits all orbit one question: who owns the unlabeled data that a self-supervised model consumed.

Sources

educational