Every mainstream multimodal assistant descends from a 2020 paper that asked a deceptively simple question: what if you treated a photograph like a sentence? The paper was "An Image is Worth 16x16 Words," from Alexey Dosovitskiy and ten colleagues at Google Research, posted to arXiv in October 2020 and published at ICLR in 2021. Its model, the Vision Transformer or ViT, is the reason the image you paste into a chatbot is billed as a block of tokens. Here is what ViT does, why it only worked at enormous scale, and how it became the eye of nearly every vision-language model shipped since.
The problem: transformers read sequences, pictures are grids
The Transformer architecture described by Vaswani and colleagues in 2017 takes a sequence of discrete tokens as input. That fits language naturally: text splits into words, or in practice into subword tokens. Images do not work that way. A 224x224 color photograph is a grid of 224 rows and 224 columns, each cell holding three color values, about 150,000 numbers in total.
Running a standard Transformer over raw pixels was impractical because self-attention scales with the square of the sequence length, and a pixel-by-pixel view of an image also has no natural token vocabulary. So for the decade before 2020, deep computer vision belonged to convolutional neural networks. A CNN bakes assumptions about pictures into its structure: nearby pixels are related, and a feature like an ear should register the same way wherever it appears in the frame. By the late 2010s the best image classifiers were large ResNet variants, and researchers who wanted attention had to graft it onto convolutions rather than replace them.
The trick: cut the picture into squares
ViT's core idea is one operation. Slice the image into a grid of fixed-size patches, 16x16 pixels in the standard configuration. A 224x224 image yields a 14x14 grid, which is 196 patches. Each patch is flattened into a vector and pushed through a single learned linear projection that maps it into the model's embedding width. That is the entire "tokenizer" for images: every patch becomes exactly one token, treated the same way as a word is treated in an NLP model.
The sequence of 196 patch embeddings gets positional embeddings added, because attention itself is order-agnostic and the model otherwise cannot tell whether a patch came from the top left or the bottom right. The ViT paper uses simple learnable 1D positional embeddings, one per grid position. A special classification token, a learned vector prepended to the sequence, rides along; after the stack of Transformer encoders processes everything, that token's final state feeds the classifier head.
From there it is an off-the-shelf text Transformer encoder. No convolution, no pooling, no vision-specific machinery. The paper's two flagship configurations were ViT-Base with 86 million parameters (12 layers, embedding width 768, 12 attention heads) and ViT-Large at roughly 300 million.

The catch: it only worked on internet-scale data
The first result was discouraging. Trained and evaluated on ImageNet alone, the 1.3 million image dataset that anchors academic computer vision, ViT finished a few percentage points behind ResNet models of comparable size. The authors' explanation is the paper's real finding: Transformers lack the inductive biases built into CNNs, such as translation equivariance and locality, so they generalize poorly when data is scarce.
Change the training set and the picture inverts. Pre-train on 14 million images (ImageNet-21k) or Google's internal JFT-300M, a labeled collection of roughly 300 million images, and the plain ViT matches or beats the best convolutional networks on ImageNet, CIFAR-100 and the VTAB benchmark suite while requiring substantially less pre-training compute. The paper's summary sentence became a slogan: large scale training trumps inductive bias.
That reframed the debate. The cost of ViT's missing structure was not architectural, it was data. Give the model enough labeled pictures and hand-designed priors stop mattering, which is the same argument reinforcement-learning scaling and the Bitter Lesson made for language, arriving in vision two years later.
What came after: scaling the eye
Once ViT proved the interface, labs applied the familiar LLM scaling playbook to images.
"Scaling Vision Transformers" (Zhai and colleagues, June 2021) refined the recipe and trained a ViT with two billion parameters, then the state of the art on ImageNet at 90.45% top-1, with strong few-shot transfer (84.86% with only ten examples per class). "Scaling Vision Transformers to 22 Billion Parameters" (Dehghani and colleagues, February 2023) went further: ViT-22B, used as a frozen feature extractor, reaches 89.5% on ImageNet, and distilling from it gives a small ViT-B student 88.6%, better than any model its size at the time. The authors described the result as evidence for LLM-like scaling in vision. Getting a 22-billion-parameter ViT to train stably required QK normalization in attention, a stability trick already familiar from large language model training.
Meanwhile CLIP, released by OpenAI in February 2021, trained image and text encoders jointly on 400 million web-scraped image-text pairs. Its best variant was a ViT-L/14 at 336-pixel input, and it could classify new tasks zero-shot, reaching 76.2% top-1 on ImageNet without ever training on ImageNet labels. CLIP mattered commercially because it handed language models a pretrained eye that already spoke text.
How ViT became the eye of language models
The dominant multimodal recipe for the past several years is literally a ViT bolted to a Transformer decoder. LLaVA (Liu and colleagues, April 2023) connects a frozen CLIP ViT-L/14 visual encoder to the Vicuna language model through one learned projection layer, turning image tokens into embedding-space tokens the language model consumes. Meta's Segment Anything Model (April 2023) uses an MAE-pretrained ViT-H as its image encoder. The lineage runs straight into today's assistants: Qwen2-VL (September 2024) keeps the patch-token paradigm but adds what its authors call naive dynamic resolution, letting one image produce a variable number of visual tokens instead of forcing everything into a fixed grid, and fuses positional information across text, images and video with multimodal RoPE.
That is why your multimodal API bill reads the way it does. An image is not magic; it is roughly one token per patch, a fixed 256-ish sequence in the early designs and a dynamic count in newer ones. The tokenizer for pictures was invented in 2020 and most of the industry has not needed a fundamentally different one since.

What it means in practice
Three lessons from the ViT story still carry weight:
- Uniform architectures scale better than specialized ones. Teams stopped maintaining separate vision and language stacks because both became the same Transformer block.
- Missing inductive bias is a data problem, not a death sentence. ViT lost to ResNet on small datasets and won on large ones, exactly as the authors predicted both ways.
- Interface tricks outperform clever internals. Cutting images into squares added nothing intellectually deep, but it let twenty years of NLP scaling research transfer wholesale into vision.
ViT did not make convolutions obsolete; they still dominate efficient edge inference and dense prediction workloads. But as the standard way to turn a picture into something a language model can read, the patch sequence has no serious competitor yet.
Sources
- Dosovitskiy et al., "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" (2020): https://arxiv.org/abs/2010.11929
- Zhai et al., "Scaling Vision Transformers" (2021): https://arxiv.org/abs/2106.04560
- Dehghani et al., "Scaling Vision Transformers to 22 Billion Parameters" (2023): https://arxiv.org/abs/2302.05442
- Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (CLIP, 2021): https://arxiv.org/abs/2103.00020
- Liu et al., "Visual Instruction Tuning" (LLaVA, 2023): https://arxiv.org/abs/2304.08485
- Kirillov et al., "Segment Anything" (2023): https://arxiv.org/abs/2304.02643
- Bai et al., "Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution" (2024): https://arxiv.org/abs/2409.12191
- Vaswani et al., "Attention Is All You Need" (2017): https://arxiv.org/abs/1706.03762
- Related on llms.blog: "A Picture Is Still 1,200 Tokens": https://www.llms.blog/posts/a-picture-is-still-1-200-tokens-how-multimodal-inputs-get-billed-in-llm-apis
