Subscribe
AI
8 min read

What Is TF-IDF? The 1972 Formula That Still Runs Search

Every retrieval system eventually has to answer the same question: given a pile of text and a query, which pieces of the pile matter? Modern AI answered it with embeddings and vector databases. But the scoring idea underneath most keyword search, and much of the keyword half of today's hybrid RAG pipelines, is a formula from 1972. It fits on one line of a whiteboard, it needs no training data and no GPU, and it still decides what Elasticsearch returns when you type something into a search box.

What Is TF-IDF? The 1972 Formula That Still Runs Search

Every retrieval system eventually has to answer the same question: given a pile of text and a query, which pieces of the pile matter? Modern AI answered it with embeddings and vector databases. But the scoring idea underneath most keyword search, and much of the keyword half of today's hybrid RAG pipelines, is a formula from 1972. It fits on one line of a whiteboard, it needs no training data and no GPU, and it still decides what Elasticsearch returns when you type something into a search box.

This is the story of TF-IDF: what it computes, where it came from, why it lost the search engine rankings battle in the 2020s, and why it refuses to leave the building.

The formula in one paragraph

TF-IDF stands for term frequency times inverse document frequency. It assigns a weight to every word in every document in a collection, and the weight answers a specific question: how much does this word say about this document?

The two factors pull in opposite directions.

Term frequency (TF) is a count. If the word "pneumonia" appears three times in a medical report, the report is probably about pneumonia. Naive versions use the raw count; most practical versions compress it, because a document that repeats a word ten times is not ten times more relevant than one that repeats it twice.

Inverse document frequency (IDF) is a rarity bonus. It divides the total number of documents in the collection by the number of documents containing the term, then takes the logarithm of that ratio. Common words like "the" or "model" appear almost everywhere, so their IDF is close to zero and they contribute almost nothing. Rare words that concentrate in a few documents get a large IDF.

Multiply them, and the intuition falls out: a word matters when it shows up often in this document but rarely in the collection overall.

A rare word outweighing a pile of common ones

A worked example

Take a corpus of 10,000 documents. The word "pneumonia" appears in 50 of them, so its IDF is log10(10,000 / 50) = 2.30. The word "infection" appears in 3,000 documents, giving it an IDF of log10(10,000 / 3,000) = 0.52. The word "the" appears in 9,000 documents, giving it 0.05.

Now consider a document that uses "pneumonia" twice and "infection" five times. TF-IDF scores "pneumonia" at 2 x 2.30 = 4.60 and "infection" at 5 x 0.52 = 2.61. Five mentions of the generic word lose to two mentions of the specific one. That is the entire trick, and it is why a search engine can rank a corpus of millions of documents without understanding a word of it.

Who wrote it down first

The inverse-document-frequency half traces to a single 1972 paper by Karen Spärck Jones, A Statistical Interpretation of Term Specificity and Its Application in Retrieval, published in the Journal of Documentation. Spärck Jones called it "term specificity": an empirical observation that words occurring in fewer documents do more work distinguishing them. She came from medical-literature retrieval work in Cambridge, and her heuristic survived five decades of search engines largely unmodified.

The rest came from Gerard Salton's SMART project at Cornell, which built the vector space model of information retrieval in the 1960s and 1970s: represent each document as a vector of term weights, represent the query the same way, and rank documents by the cosine of the angle between them. SMART's experiments adopted idf as the standard term weight, and Salton and Chris Buckley's 1988 survey Term Weighting Approaches in Automatic Text Retrieval systematized the tf x idf product and its many variants (log-scaled TF, smoothed IDF, length normalization). The textbook everyone still uses, Introduction to Information Retrieval by Manning, Raghavan and Schütze (2008), devotes a full chapter to the family.

One detail worth knowing: for nearly thirty years nobody could say why IDF works. It was a heuristic with excellent benchmarks. The first clean information-theoretic derivation came from Akiko Aizawa's 2000 SIGIR paper, An Information-Theoretic Perspective of tf-idf Measures, which showed that tf-idf sums to the mutual information between the document set and a term set. Stephen Robertson's 2004 review, Understanding Inverse Document Frequency: On Theoretical Arguments for IDF, assessed the competing justifications and found the solid ones in the probabilistic retrieval tradition. The math caught up only after the formula was already deployed everywhere.

What TF-IDF is bad at

The weaknesses are exactly as old as the strengths, and they explain the next two decades of retrieval research.

  • No synonyms. A document about "heart attacks" scores zero for the query "myocardial infarction." TF-IDF only sees literal term overlap.
  • No polysemy handling. "Python" gets the same weight whether the document is about snakes or programming.
  • Collection dependence. Every IDF value is a property of the corpus, not the language. Move the same document into a different collection and all its weights change.
  • Bag-of-words amnesia. Word order, phrases and negation are invisible. "not good" and "good" score identically for the term "good."
  • Length artifacts. Raw counts favor long documents, which is why practical implementations always add compression and normalization steps.

The classical fix for the first two was relevance feedback and query expansion; the modern fix is neural. But note that the last two problems, collection statistics and length normalization, were never really "solved," only inherited carefully by everything that came next.

The heir: BM25, the same idea with a steering wheel

The formula that actually runs most keyword search today is not textbook TF-IDF. It is BM25, "Best Match 25", the twenty-fifth in a series of probabilistic ranking models (BM1 through BM25) developed by Stephen Robertson, Karen Spärck Jones and colleagues at City University London's Okapi system in the late 1980s and early 1990s. Robertson and Zaragoza's 2009 survey "The Probabilistic Relevance Framework: BM25 and Beyond" is the standard reference.

BM25 keeps the Spärck Jones insight, a log-ratio IDF, and fixes the two mechanical flaws of raw TF-IDF:

  1. TF saturation. Term frequency gains diminishing returns, controlled by a parameter k1 (typically 1.2). The fifth mention of "pneumonia" adds less than the first.
  2. Document length normalization. A parameter b (typically 0.75) discounts long documents that accumulate term counts by sheer volume.

So BM25 is best understood as TF-IDF's great-grandchild: same scoring philosophy, better calibration. That lineage is production infrastructure. Apache Lucene, the engine inside Elasticsearch and OpenSearch, made BM25 its default similarity in version 6.0 in 2016, replacing the TF-IDF-derived "Classic" similarity it had used since the 1990s. Every default full-text search field in Elasticsearch today is a BM25 index.

The embedding interlude

In 2020, Dense Passage Retrieval (Karpukhin et al., Facebook AI and Stanford) declared the sparse statistical models, TF-IDF and BM25, the "de facto method" and then beat them. Trained dual encoders mapped questions and passages into dense vectors, and the dense retriever outperformed a strong Lucene-BM25 baseline by 9 to 19 points absolute in top-20 retrieval accuracy across open-domain question answering datasets. The result used only a few thousand training examples, and it helped launch the current era: embeddings, vector databases, and semantic search as the default retrieval story for LLM applications.

By 2023 the conventional product wisdom was that TF-IDF's lineage was a legacy artifact, useful only when someone searched for an exact string. That turned out to be premature.

The lexical comeback in RAG

Two things happened. First, practitioners building RAG systems discovered that dense retrieval has a soft spot for exactly the queries that matter most in enterprise corpora: part numbers, error codes, tickers, function names, rare spellings. A bi-encoder can generalize "heart attack" to "myocardial infarction"; it can also quietly generalize ERR-4019-XZ into a semantically plausible but wrong chunk. Inverted indexes do not generalize at all, which is their strength.

Second, the research line did not choose sides. SPLADE (Formal et al., SIGIR 2021, NAVER Labs Europe) uses BERT not to produce a dense vector but to learn term weights in the original vocabulary space, expansion included, explicitly in the tf-idf tradition: estimate how important each word is, store it in a classic inverted index. The 2021-era learned-sparse wave and the dense wave converged into hybrid retrieval, keyword scores and embedding scores fused (usually reciprocal rank fusion), now the default architecture recommendation for production RAG.

The number to remember is Anthropic's. Its 2024 Contextual Retrieval experiments reported that adding a lexical BM25 index on top of contextual embeddings cut top-20 chunk retrieval failure from 3.7% to 2.9%, and that the combined system reduced failure rates 49% against a plain embedding baseline (5.7% to 2.9%), reaching 67% with reranking. The lexical half, direct descendant of a 1972 term-specificity heuristic, is carrying a meaningful share of that improvement.

Keyword and semantic lanes merging into one ranked list

Why the formula still deserves a place in your head

Three practical reasons.

It is the cheapest retrieval that exists. No model to run, no GPU to rent, no embedding API bill. Indexing is counting, and querying is arithmetic on posting lists. When teams discover their $2,000-a-month vector database is being used mostly for exact product-code lookups, TF-IDF lineage scoring handles the job at near-zero marginal cost.

It is auditable. An embedding similarity of 0.81 means nothing you can explain. A ranking that says "this document matched your rarest query term four times" is legible to a customer support lead. Learned-sparse hybrids like SPLADE keep the term-level weights for the same reason.

It is the conceptual ancestor you keep re-meeting. Inverted indexes, term weighting, saturation curves and collection statistics are the grammar of every retrieval tool an agent calls today, and of the data-selection ideas (term importance, rarity-based filtering) that feed training-pipeline work like deduplication and quality classification. Understanding why a rare term gets a higher weight is most of what you need to reason about why retrieval-augmented systems fail and which retrieval layer you would change.

Karen Spärck Jones wrote a heuristic in 1972, could not fully justify it theoretically, and watched it become the scoring philosophy of every search engine built since. The log-ratio of document counts is the most durable piece of arithmetic in the information age. The hype cycles will keep claiming it is dead. The indexes keep disagreeing.

Sources

AILLMeducational