Subscribe
Education
6 min read

What Are Embeddings? How Language Models Turn Words Into Numbers

Every retrieval system, recommendation engine, and RAG pipeline starts with the same move: turn text into a list of numbers. That move is called an embedding, and it is older and weirder than most production descriptions suggest. This piece explains what an embedding actually is, where the idea came from, and what the geometry does and does not buy you. The idea predates the neural networks The core intuition is from linguistics, not machine learning. In 1957, the British linguist J.R. Firth

Mid-century brochure diagram of the king minus man plus woman equals queen vector parallelogram, titled What Are Embeddings?
Mid-century brochure diagram of the king minus man plus woman equals queen vector parallelogram, titled What Are Embeddings?

Every retrieval system, recommendation engine, and RAG pipeline starts with the same move: turn text into a list of numbers. That move is called an embedding, and it is older and weirder than most production descriptions suggest. This piece explains what an embedding actually is, where the idea came from, and what the geometry does and does not buy you.

The idea predates the neural networks

The core intuition is from linguistics, not machine learning. In 1957, the British linguist J.R. Firth wrote that "you shall know a word by the company it keeps." The meaning of a word, in this view, is not a definition stored somewhere but a pattern of contexts: which words appear near it, how often, in what roles. A related argument came from Zellig Harris in 1954, who laid out a distributional theory of structure years before anyone could compute it at scale.

That intuition became computable once large text corpora existed. If meaning is context behavior, then you can measure context behavior statistically, and words with similar behavior can be represented by similar numbers. Every embedding method since is a variation on that sentence.

Word2vec made it arithmetic

The method that launched the modern era was word2vec, published by Tomas Mikolov and colleagues at Google in 2013. It trained a shallow neural network on a corpus of roughly a billion words with one of two objectives: predict a word from its surrounding context (CBOW) or predict the context given a word (skip-gram). The network itself was disposable. The product was the learned weight matrix, which assigned each word a vector of 50 to 500 numbers in the paper's experiments.

The striking result was that these vectors supported arithmetic. In a follow-up paper, Mikolov et al. showed that patterns like "man is to king as woman is to queen" could be reproduced by vector operations: take the king vector, subtract man, add woman, and the nearest result is queen. They built a test set of 8,869 semantic and 10,675 syntactic analogies to measure the effect. A 2024 analysis of the same trick adds a dry caveat: if you do not exclude the input words from the nearest-neighbor search, the closest vector to king minus man plus woman is king itself. The arithmetic works, but it is more fragile than the pop-science version suggests.

Diagram: context marks feeding into a central point that outputs a short numeric vector, captioned Learning vectors from context

Stanford's GloVe (Pennington, Manning and Gupta, 2014) offered a different route to the same destination: count word co-occurrences across the whole corpus first, then fit vectors that reproduce those counts. Word2vec learns from local windows one example at a time; GloVe learns from a global statistics matrix. Both produce static embeddings, one vector per word form, which means "bank" gets the same coordinates whether you mean a river bank or a savings account.

The geometry, and why people use cosine

An embedding is a point in a high-dimensional space. Relatedness between texts becomes a geometric relation between points, and the standard measure is cosine similarity: the cosine of the angle between two vectors, which ignores their magnitude and compares only their direction. Most embedding APIs normalize their outputs, so cosine similarity and dot product coincide in practice.

The honest caveat is that high-dimensional spaces behave in ways the flat diagrams do not capture. Kawin Ethayarajh's 2019 analysis of BERT, ELMo, and GPT-2 found that contextualized embeddings are not isotropic: all word vectors cluster in a narrow cone of the space rather than spreading evenly in every direction. That geometry inflates the cosine similarity between any two random words, which is part of why raw similarity scores can look meaningless until corrected. The same paper found that on average less than 5% of the variance in a word's contextual vectors can be explained by a single static embedding for that word, a quantitative argument for why context-sensitive models displaced word2vec-style ones.

Embeddings inside a language model

In a transformer, "embedding" names a specific component: a learned lookup table with one row per vocabulary token. At input, each token id indexes a row of that matrix and becomes the model's starting representation of the token. Nothing about meaning is hard-coded there; the geometry is a byproduct of the training objective. GPT-2 also ties this input embedding matrix to the output projection that produces next-token probabilities, sharing the same weights in both directions, which saves parameters and couples the input and output spaces.

This is the distinction worth keeping straight. The embedding layer of an LLM is context-free by construction. Attention and the rest of the network then produce contextualized representations, where "bank" next to "river" and "bank" next to "mortgage" end up at different points. The static-versus-contextual split traces directly to the word2vec era and still shapes how retrieval systems are built.

Embeddings as a product

Dedicated embedding models map whole texts to single vectors for use in search and retrieval infrastructure. OpenAI's text-embedding-3 generation, released in January 2024, produces vectors with up to 3,072 dimensions and reported score jumps from 61.0 to 64.6 average on the MTEB English benchmark and from 31.4 to 54.9 on the MIRACL multilingual retrieval benchmark relative to its ada-002 predecessor. The models are trained so that trimming dimensions from the end of a vector degrades quality slowly (a technique from Matryoshka representation learning), letting teams trade storage and latency against accuracy without retraining.

Sentence-level embedding is a separate trick from word-level embedding. Sentence-BERT (Reimers and Gurevych, 2019) fine-tuned BERT so that a single fixed-length vector represents an entire sentence, making semantic search over millions of texts practical with ordinary nearest-neighbor infrastructure instead of slow pairwise model passes. Multi-Sentence Text Embedding Benchmark (MTEB), introduced in 2022, is the standard leaderboard these systems are measured on.

The chunk-embedding problem is where RAG pipelines feel the static-versus-contextual legacy directly. If you cut a long document into small chunks before encoding them, pronouns and references that only resolve earlier in the document get lost, and the chunk vectors silently misrepresent meaning. Jina AI's late chunking method (2024) inverts the order: encode the full long-context sequence first, then pool chunk-level vectors from the contextualized representations, so each chunk embedding keeps the context it needs.

Where the vectors actually get used

  • Semantic search and RAG: query and corpus chunks are embedded with the same model, and retrieval reduces to nearest-neighbor lookup.
  • Clustering and deduplication: groups of near-duplicate documents fall within a distance threshold of each other, which makes embeddings a cheap redundancy filter at corpus scale.
  • Classification features: a fixed embedding turns variable-length text into a fixed-width input for a small classifier.
  • Recommendations: users and items embedded in one space means "similar to this" is a lookup, not an editorial judgment.
Diagram: a query vector linked by a dotted line to the nearest cluster of points on a grid, captioned Similarity as distance

What embeddings do not do is store facts or guarantee similarity judgments that match human intuition. The vectors are a geometry learned from co-occurrence, useful precisely because it compresses, and error-prone for the same reason.

Sources

EducationLLM