Subscribe
Subscribe
AI
8 min read
axc
axc

What Is RAG? Retrieval-Augmented Generation Explained in Plain English (2026)

RAG lets an AI model look up relevant documents before it answers. A plain-English guide to how retrieval-augmented generation works, when you need it, where it fails, and the hosted RAG tools from OpenAI, Google, and Anthropic as of October 2026.

What Is RAG? Retrieval-Augmented Generation Explained in Plain English (2026)

Retrieval-augmented generation, or RAG, is the most common way to make an AI model answer questions about information it was never trained on: your company handbook, a product manual, last week's support tickets, or a folder of contracts. Instead of hoping the model remembers the right facts, a RAG system looks them up first and hands them to the model along with the question.

If you use ChatGPT with uploaded files, a customer support bot that quotes help articles, or an internal "ask the docs" tool at work, you have almost certainly used RAG. This guide explains what it is, how it works step by step, when you need it, and where it goes wrong, in plain English. It is written for people who are new to the term. Developers will find links to the deeper material at the end.

RAG in one sentence

AWS defines RAG as "the process of optimizing the output of a large language model, so it references an authoritative knowledge base outside of its training data sources before generating a response." Put more simply: retrieve the relevant text, add it to the prompt, then generate the answer.

A useful comparison is an open-book exam. A model without RAG answers from memory. A model with RAG gets to look up the right pages first, and a good system tells you which pages it used.

Where the term comes from

The name comes from a 2020 research paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by Patrick Lewis and 11 co-authors, first posted to arXiv on May 22, 2020. The paper describes models that combine "parametric memory" (what a model stores in its weights) with "non-parametric memory" (in their case, a dense vector index of Wikipedia that the model searches with a neural retriever).

Meta's announcement from September 28, 2020 made the practical point that still drives RAG's popularity: unlike a plain pretrained model, a RAG system's knowledge "can be easily altered or even supplemented on the fly." You update the documents, not the model.

Today the word covers a much broader family of systems than the original paper. Most production RAG does not fine-tune anything. It pairs an off-the-shelf model with a search step.

Why RAG exists

Large language models have three limits that RAG addresses:

  • Knowledge cutoff. A model only knows what was in its training data. It does not know your internal documents or anything that happened after training.
  • Hallucination. When a model does not know an answer, it can produce a confident, wrong one. Giving it the relevant source text reduces that, though it does not eliminate it. Our AI hallucinations explainer covers why models make things up.
  • No sources. A plain model cannot point to where a fact came from. RAG systems can show the passages they used, so a person can check.

The 2023 RAG survey by Yunfan Gao and colleagues lists the same problems ("hallucination, outdated knowledge, and non-transparent, untraceable reasoning processes") as the reasons RAG became a standard technique.

How RAG works, step by step

Five-step diagram of a RAG pipeline: split documents, embed and index, retrieve top chunks, add to the prompt, answer with sources

A typical RAG system has two phases. The first runs once, ahead of time. The second runs every time someone asks a question. Anthropic's Contextual Retrieval write-up describes the standard version this way.

Ahead of time (indexing):

  1. Split the documents into chunks. Long documents are cut into smaller pieces, "usually no more than a few hundred tokens," per Anthropic. A token is roughly a word or part of a word.
  2. Turn each chunk into an embedding. An embedding model converts each chunk into a list of numbers (a vector) that represents its meaning. Chunks about similar topics end up with similar vectors. Some small embedding models, such as EmbeddingGemma 2, can run on a phone or laptop.
  3. Store the vectors in a searchable index. This is usually a vector database, which can quickly find the chunks closest in meaning to a new piece of text.

On every question (retrieval and generation):

  1. Retrieve. The user's question is turned into an embedding too, and the system finds the chunks most similar to it. Many systems also run a keyword search (often an algorithm called BM25) to catch exact terms such as product codes or error numbers that meaning-based search can miss.
  2. Augment. The top chunks are pasted into the prompt alongside the question, usually with an instruction such as "answer using only the context below."
  3. Generate. The model writes the answer from that context. Better systems also return citations to the chunks used.

That is the whole idea. Everything else in RAG engineering is about making the retrieve step find the right text more often, and making the generate step stick to it.

A simple example

Say an employee asks an HR assistant, "How many days of parental leave do I get?"

  • Without RAG, the model guesses from general knowledge about typical leave policies. It might be wrong for this company.
  • With RAG, the system searches the company's policy documents, finds the paragraph on parental leave, and gives it to the model. The model answers from that paragraph and links to it.

If the policy changes next month, someone updates the document and re-indexes it. No retraining is needed.

RAG vs fine-tuning vs a long prompt

RAG is one of three common ways to give a model knowledge it lacks:

  • Put everything in the prompt. If your material is small, you can skip retrieval and paste all of it in. Anthropic's guidance is that if a knowledge base is "smaller than 200,000 tokens (about 500 pages of material)," you can include the entire thing in the prompt with "no need for RAG or similar methods," and prompt caching makes that cheaper and faster.
  • Fine-tuning. Training a model further on your data changes its behavior, tone, and format well. It is a poor way to keep facts current, because every update means another training run, and the model still cannot cite where a fact came from.
  • RAG. Best when the material is large, changes often, or must be cited. It is the usual choice for document search, help centers, and internal knowledge bases.

These are not exclusive. Many products use a fine-tuned or carefully prompted model on top of a RAG pipeline.

Where RAG goes wrong

RAG is simple to describe and hard to get right. The common failure points:

  • Retrieval misses. If the search step does not find the right chunk, the model cannot use it. This is the most common failure, and it is often invisible, because the model may still produce a fluent answer from the wrong chunks.
  • Chunks lose context. A chunk that says "revenue grew 3% over the previous quarter" is useless if it does not say which company or which quarter. Anthropic's Contextual Retrieval post calls this the main weakness of traditional RAG: "traditional RAG solutions remove context when encoding information." How you split documents matters a great deal; our comparison of chunking strategies covers the options.
  • Too much context. Stuffing in more chunks is not always better. The paper Lost in the Middle (Liu et al., 2023) found that model performance "is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts."
  • The model ignores the sources. Even with the right text in the prompt, a model can blend in its own assumptions. Asking for citations and checking them helps.

How much better retrieval helps

Bar chart of Anthropic's top-20 chunk retrieval failure rates: standard embeddings 5.7%, contextual embeddings 3.7%, plus contextual BM25 2.9%, plus reranking 1.9%

Anthropic's September 19, 2024 experiments give a concrete sense of how much the retrieval step can improve. It measured how often the relevant passage failed to appear in the top 20 retrieved chunks:

  • Adding a short, model-written context note to each chunk before embedding it ("Contextual Embeddings") cut the failure rate by 35%, from 5.7% to 3.7%.
  • Adding the same treatment to keyword search ("Contextual BM25") cut it by 49%, to 2.9%.
  • Adding a reranking step, where a second model re-scores the retrieved chunks (Anthropic tested the Cohere reranker), cut it by 67%, to 1.9%.

These are Anthropic's own numbers on its own test sets, not a universal benchmark, but they show the pattern most teams find: the biggest gains usually come from better search, not a bigger model. For more on mixing keyword and meaning-based search, see our deep dive on hybrid search and reranking.

Common RAG variants you will hear about

  • Hybrid search. Combines embedding search with keyword search, then merges the results. Now a common default.
  • Reranking. Retrieves a larger set of candidates, then uses a second model to pick the best few.
  • GraphRAG. Builds a knowledge graph of entities and relationships from the documents and retrieves through that graph. Microsoft's GraphRAG project popularized the idea after its July 2024 release; as of October 2026 its README says the project "is largely in maintenance mode" and will not add new features. Our GraphRAG framework comparison covers the alternatives.
  • Agentic RAG. Instead of one fixed search, an AI agent decides what to search for, runs several searches, and checks whether it has enough before answering.

Can I use RAG without building it?

Yes. The major AI platforms now offer hosted retrieval, so you can upload files and let the provider handle chunking, embedding, and search:

  • OpenAI offers file search, a tool in its Responses API that lets models search uploaded files "through semantic and keyword search" using vector stores that OpenAI manages.
  • Google offers a File Search tool in the Gemini API. Per Google's documentation as of October 2026, you pay for embeddings when files are indexed, while storage and query-time embeddings are free, and retrieved text is billed as regular input tokens.
  • Anthropic offers citations in the Claude API, which return "the exact passages that support each claim" when Claude answers from documents you provide.
  • Microsoft documents RAG patterns for Azure AI Search.

If you want to build your own, the open-source frameworks LlamaIndex and LangChain both have RAG tutorials, and Hugging Face's Advanced RAG cookbook walks through a full pipeline with open models. To measure whether your system actually retrieves the right material, Ragas is a widely used open-source evaluation library.

How to learn more

The short version

RAG means looking up relevant text and giving it to the model before it answers. It is the standard way to make AI tools work with private, current, or citable information without retraining a model. It works as well as its search step does: most of the effort in a good RAG system goes into splitting documents sensibly, finding the right chunks, and checking that the answer actually uses them.

Sources

AILLMArchitectureRAG

Written by

axc
axc

1,219 articles

Edited by the llms.blog desk.

More in RAG

All RAG →

Popular this week