Every architecture explainer on this site eventually runs into the transformer. Before it, and still underneath parts of it today, sits an older idea: the convolution. It is the reason computer vision worked at all, and it is quietly back in the language-model conversation.
The idea in one paragraph
A convolutional layer does not look at an image the way a fully connected layer does. Instead of wiring every pixel to every neuron, it takes a small learnable filter, say 3×3 pixels wide, and slides it across the whole input. At each position the filter produces one number: how strongly the local patch matches the pattern the filter has learned. Stack enough of these sliding filters and you get a network whose first layers detect edges, whose middle layers detect textures and parts, and whose last layers detect objects.

Two properties make this useful. Locality: each neuron only ever sees a small window, so the network starts with the assumption that nearby pixels matter together. Weight sharing: the same filter is reused at every position, which is why one edge detector costs 28 parameters instead of millions. Stanford's CS231n course notes put concrete numbers on this: for a 224×224×3 image connected to a 1×1×4 hidden volume, a fully connected layer would need about 6 million parameters. The same four features as 3×3×3 convolution filters (plus biases) cost 28.

Weight sharing buys more than memory. Because the same filter slides everywhere, a pattern the network learns in one corner of an image is automatically recognized in every other corner. Practitioners call this translation equivariance, and it is the free gift that hand-engineered vision pipelines of the 1990s could never replicate.
From cat visual cortex to mail sorting
The design was not invented from math. In the late 1950s and early 1960s, David Hubel and Torsten Wiesel recorded from cat visual cortex and found two kinds of neurons: "simple cells" that fired for an oriented edge at a specific spot, and "complex cells" that fired for the same edge almost anywhere in a local region. The S-cell/C-cell pair is the direct ancestor of today's convolution/pooling stack.
In 1980, Japanese researcher Kunihiko Fukushima built the neocognitron, a multilayer model with alternating simple and complex cell layers that learned by self-organization rather than backpropagation. It recognized patterns unaffected by shifts in position, which is convolution's defining behavior, but it could not be trained end-to-end on a task.
Yann LeCun closed that gap. In 1989 he applied backpropagation to a convolutional network that read handwritten digits from the US Postal Service with roughly 1% error on the digits themselves. The 1998 follow-up paper, "Gradient-Based Learning Applied to Document Recognition," described LeNet-5, the template every modern vision network still follows: alternating convolution and subsampling layers, trained end-to-end from raw pixels to final class, no hand-designed features anywhere. Variants of these systems went into production reading handwritten ZIP codes at USPS in the 1990s.
Then came the long pause. CNNs were the standard tool for OCR and small recognition tasks, but the wider field treated hand-engineered features plus classical classifiers as the serious approach. That lasted until 2012.
2012: the year the argument ended
The ImageNet benchmark, released in 2009 and contested annually from 2010, threw 1.2 million labeled images across 1,000 categories at whatever method entrants liked. At the 2012 competition, a team from the University of Toronto, Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, entered a convolutional network now known as AlexNet. It scored 15.3% top-5 error. The runner-up, built from hand-engineered features and classical classifiers, scored 26.2%. In one afternoon the error gap that had taken vision researchers a decade to grind down by a point or two was cleared by more than 10 points.
AlexNet was recognizably LeNet, scaled: eight layers, roughly 60 million parameters, ReLU activations, dropout, and a training run of five to six days on two consumer-grade Nvidia GTX 580 GPUs. The lesson the industry took was not the architecture, which was already nine years old. The lesson was data plus GPUs plus a deep convolutional stack, and most research money moved accordingly.
The years after were a depth race. VGGNet in 2014 showed that long stacks of tiny 3×3 convolutions beat AlexNet-style fat filters. Microsoft Research's ResNet won ILSVRC 2015 with 152 layers and 3.57% top-5 error, below the roughly 5.1% error the ImageNet creators had measured for trained human labelers. At some point between 2015 and 2020, reading a picture became solved enough to be a commodity.
The transformer took the crown, then the convolution crept back
In 2020, Google's Vision Transformer paper asked what would happen if you treated 16×16 image patches as tokens and ran plain attention with no convolutions at all. With enough pre-training data it beat CNNs, and by the mid-2020s the dominant image encoders inside vision-language models were ViT-style transformers. For a while, the convolution looked like the architecture it was taught in history classes: important, retired.
Two lines of work complicated that story. First, Meta's 2022 ConvNeXt project modernized a pure convolutional network using transformer-era recipes (larger batches, patchified inputs, layer norm, GELU) and matched Swin transformers on ImageNet, a reminder that much of what people attributed to attention was actually about scale and training discipline.
Second, and more relevant here, convolutions reappeared inside sequence models built to avoid attention's quadratic cost. The state-space model lineage (S4, Mamba) describes its input-output behavior as a 1D convolution with a structured kernel; Mamba blocks include a short causal 1D convolution as a standard component. Meta's Hyena Hierarchy work went further, replacing attention in language models with very long depthwise convolutions whose kernels are generated on the fly by small neural networks. The reason is the same property the 1980s models leaned on: one shared kernel scanned across a sequence costs far less than quadratic attention, as long as you are willing to give up attention's ability to build a fresh, query-dependent pattern at every step.
That is the real distinction between the two mechanisms. A convolution applies a learned filter uniformly across positions; attention decides, per token, where to look. Attention wins when the relevant context is unpredictable; convolutions win when locality and budget dominate, which describes a lot of language and most of audio. Modern hybrid models keep both, and the conv half is doing work that traces directly back to Fukushima's S-layers.
Key numbers
- LeNet (1998): end-to-end CNN; USPS digit recognition with ~1% digit error (LeCun et al., Proceedings of the IEEE)
- AlexNet (2012): 15.3% vs 26.2% top-5 error at ILSVRC; ~60M parameters; 2 GTX 580 GPUs, 5-6 days
- ResNet (2015): 152 layers, 3.57% top-5, under the ~5.1% human estimate
- ViT (2020): image patches + attention, no convolutions needed at scale
- ConvNeXt (2022): pure convolutions, transformer-era training, matches Swin
- Mamba (2023) / Hyena (2023): causal 1D convolutions inside attention-free sequence models
The convolution is not the future of AI. It is the plumbing: invisible, old, and load-bearing in more places than the coverage suggests.
Sources
- Stanford CS231n, Convolution Neural Networks for Visual Recognition (convolution math, parameter counts, weight sharing): https://cs231n.github.io/convolutional-networks/
- Hubel and Wiesel on simple/complex cells, summarized in a review of the cortical model lineage: https://www.frontiersin.org/journals/computational-neuroscience/articles/10.3389/fncom.2023.1282828/full
- Fukushima, "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position" (1980): https://pubmed.ncbi.nlm.nih.gov/7370364/
- LeCun, Bottou, Bengio, Haffner, "Gradient-Based Learning Applied to Document Recognition" (1998): https://www.doc.ic.ac.uk/~ndm/supervision/DATASETS/lecun-98.pdf
- LeNet history and USPS deployment: https://en.wikipedia.org/wiki/LeNet
- Krizhevsky, Sutskever, Hinton, "ImageNet Classification with Deep Convolutional Neural Networks" (NeurIPS 2012): https://proceedings.neurips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
- He et al., "Deep Residual Learning for Image Recognition" (2015): https://arxiv.org/abs/1512.03385
- Simonyan and Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition" (2014): https://arxiv.org/abs/1409.1556
- Dosovitskiy et al., "An Image is Worth 16x16 Words" (2020): https://arxiv.org/abs/2010.11929
- Liu et al., "A ConvNet for the 2020s" (2022): https://arxiv.org/abs/2201.03545
- Gu and Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces" (2023): https://arxiv.org/abs/2312.00752
- Poli et al., "Hyena Hierarchy: Towards Larger Convolutional Language Models" (2023): https://arxiv.org/abs/2302.07522
