A language model only knows what it absorbed before training ended. Ask it about last week's product launch, your company's refund policy, or a filing published the day after its training cut-off, and it has to guess. Retrieval-augmented generation (RAG) is the workaround the industry settled on: instead of asking the model to recall, you look up the relevant text first and hand it to the model inside the prompt.
This post is a basic explainer. For comparisons of specific RAG frameworks and sub-architectures, we have separate pieces; this one is about what the technique is and why it exists.
The problem RAG addresses
Large language models store knowledge in their weights, which researchers call parametric memory. That memory has three stubborn properties, documented in the paper that named the technique:
- It is frozen at training time. A model "cannot easily update the knowledge stored in its parameters," as Lewis et al. put it, and can "generate nonsensical or outdated answers" when pressed on things it does not know.
- Refreshing it means retraining, which is expensive and slow relative to how often real knowledge bases change.
- It is a black box. When a model answers from its weights, there is no page to cite.
The alternative is non-parametric memory: a searchable index of documents that can be swapped, extended, or corrected without touching the model at all. RAG is the family of techniques that wires that index into generation.
Where the name comes from
The term comes from Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, published in May 2020 by Patrick Lewis and colleagues at Facebook AI Research, University College London, and New York University. Their setup paired two trained components:
- A dense retriever, built with the Dense Passage Retrieval (DPR) dual-encoder method from the same research group, which encodes questions and Wikipedia passages into vectors and retrieves the closest matches.
- A generator, a BART seq2seq model, which produces the answer conditioned on the question plus the retrieved passages.
Both were fine-tuned jointly. The paper's headline result was that the retrieval-augmented model outperformed parametric-only baselines on open-domain question answering, and generated answers that were "more factual, specific, and diverse." The paper also described how the non-parametric memory could be swapped wholesale to update knowledge, a property no fine-tune matches at comparable cost.
That 2020 design was a trained hybrid. What the industry now calls RAG is usually the training-free descendant: any retriever, any generator, glued together at inference time by prompt construction.
How a modern RAG system works
Anthropic's engineering primer describes the standard pipeline, which is representative of most production systems today.
Indexing, done once per knowledge base update:
- Split the document collection into chunks, typically no more than a few hundred tokens each.
- Run each chunk through an embedding model to get a vector that encodes its meaning.
- Store the vectors in a database that can search by semantic similarity.
Query time, on every question:
- Embed the user's question the same way.
- Search the index for the chunks closest to the question and pull the top few.
- Append those chunks to the prompt as reference material.
- The model answers using the provided text plus its own language ability.
Two refinements matter more than anything else in practice. First, embeddings alone miss exact matches: a query for a specific error code, product SKU, or statute number needs lexical search (BM25, the long-standing ranking function behind search engines) alongside vector search, with results fused together. Second, the first-pass results can be rescored by a reranker, a model that reads the query and each candidate chunk together and orders them by relevance.

Google Cloud's overview of RAG frames the payoff the way practitioners do: retrieval supplies fresh, private, or specialized data, and the model's output is "grounded" in it, which mitigates hallucination and makes answers citable and auditable. The corollary is the constraint: if the retriever returns irrelevant chunks, the generation is confidently wrong anyway. RAG quality is retrieval quality.
The weak link, and the measured fix
Because a chunked document loses the surrounding context it was cut from, retrieval frequently fails on chunks that only make sense in the parent document. Anthropic's example: a chunk reading "The company's revenue grew by 3% over the previous quarter" does not say which company or which quarter, so a query about ACME Corp in Q2 2023 may never surface it.
Their remedy, called Contextual Retrieval, prepends a short (50-100 token) model-generated explanation of where each chunk sits in its document before embedding and indexing it. In their benchmark suite, that alone cut failed retrievals by 49%, and combined with reranking it cut them by 67% (a top-20 retrieval failure rate falling from 5.7% to 1.9%). The lesson generalizes beyond their method: most RAG performance is won or lost in the indexing step, not the prompt.
When you do not need RAG
The honest caveat from the same Anthropic write-up: if your whole knowledge base fits in the model's context window, about 200,000 tokens or roughly 500 pages, you can skip retrieval and put everything in the prompt, especially with prompt caching making that cheaper between calls. Google Cloud makes the same point about long-context models: stuffing source material directly into a long window is the simpler design until the corpus or cost of tokens outgrows it.
RAG earns its complexity when the corpus exceeds the window, updates frequently, or needs per-user access permissions. Otherwise it is an extra set of moving parts between you and a working demo.
RAG versus fine-tuning
These are often posed as rivals; they do different jobs. Fine-tuning (and its parameter-efficient variants) changes the model itself, which is good for teaching style, format, or a new task, and poor for teaching a set of facts that changes weekly. RAG leaves the weights alone and changes what the model sees per request, which is exactly what you want for volatile, verifiable knowledge. Common production guidance, echoed across cloud vendors, is to use both when needed: fine-tune for behavior, retrieve for facts.

What to watch if you build one
- Measure retrieval before generation. A handful of real user questions and a check for whether the right chunk was returned will tell you more than any end-to-end benchmark.
- Chunking is a design decision, not a default. Boundary placement, chunk size, and added context all move retrieval accuracy.
- A stale index is the new model-drift. Retrieval is only as current as the last indexing run.
- Grounded does not mean correct. The model can still misread or overreach the material you hand it, which is why vendor platforms now score "groundedness" as a separate evaluation metric.
RAG is not a model architecture or a product feature. It is a plumbing decision: facts live in an index you control, and the model reads before it answers. That sentence covers 2020's research paper and 2026's enterprise chatbots alike, which is unusual longevity for a technique in this field.
Sources
- Patrick Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv:2005.11401 (May 2020, NeurIPS 2020)
- Vladimir Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering, arXiv:2004.04906 (April 2020)
- Anthropic, Introducing Contextual Retrieval
- Google Cloud, What is Retrieval-Augmented Generation (RAG)?
