Subscribe
Opinion
5 min read

Opinion: RAG Does Not End AI Hallucinations

RAG cuts AI errors but does not end them. A Stanford and Yale study found legal AI tools built on it hallucinated 17% to 33% of the time.

Olivetti-style cover reading RAG Does Not End Hallucinations, Opinion, beside a charcoal panel with teal, mustard and red blocks, on cream paper
Olivetti-style cover reading RAG Does Not End Hallucinations, Opinion, beside a charcoal panel with teal, mustard and red blocks, on cream paper

A common sales line in AI is that retrieval-augmented generation, or RAG, makes hallucinations go away. My view: RAG is useful, and it does lower errors, but it is a search problem bolted onto a writing problem. The search can miss. The writing can still go wrong. And when a RAG tool cites a source, that citation is a claim you should check, not proof that the answer is right.

What RAG actually does

RAG means the AI looks things up before it answers; our plain-English RAG explainer covers the basics. The original RAG paper from Facebook AI Research paired a language model's built-in memory with a searchable index of Wikipedia. The model finds passages, then writes an answer using them.

The results were real. The authors found that RAG models "generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline". But read how they framed the goal. They listed downsides of plain language models, including that they "may produce 'hallucinations'", and said hybrid models with a searchable memory "can address some of these issues". Some, not all. The people who named the method did not sell it as a cure.

Law is a good test, because a wrong answer with a real-looking citation can end up in a court filing. In a study first posted in May 2024, a team from Stanford and Yale tested leading AI legal research tools against what their makers had said. According to the paper, providers had described RAG as "eliminating" or "[avoid[ing]](https://arxiv.org/abs/2405.20362)" hallucinations, or as guaranteeing "hallucination-free" legal citations.

The authors built a preregistered set of over 200 legal queries. Their verdict: "the providers' claims are overstated". They did find that hallucinations "are reduced relative to general-purpose chatbots (GPT-4)". That is the fair part of the story. But the tools from LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) "each hallucinate between 17% and 33% of the time".

Number panel: legal AI research tools hallucinated between 17% and 33% of the time in a Stanford and Yale study

The tools were not equal. Lexis+ AI answered 65% of the queries accurately, the best result in the study. Ask Practical Law AI gave incomplete answers, meaning refusals or answers that did not cite support for their key claims, on more than 60% of queries. So "uses RAG" told a buyer very little about how a given product would do.

How the study counted errors matters too. A response was a hallucination if it was wrong, or if it was "misgrounded": it cited a source that did not support the claim. The authors call such errors "potentially more dangerous than fabricating a case outright, because they are subtler and more difficult to spot".

Where RAG breaks

The study says "there are several reasons that RAG is unlikely to fully solve the hallucination problem". One cause in its error analysis is naive retrieval: "failing to find the most relevant sources available". It was the most common cause behind Lexis+ AI's hallucinated answers, while reasoning errors led for the two Thomson Reuters tools. If the search step brings back the wrong documents, the writing step builds a confident answer on them.

Diagram of a RAG pipeline in three stages, find, pick and write, with failure points under each stage

This is not only a legal problem. An experience report from Deakin University looked at three RAG systems in research, education and biomedicine and listed seven failure points. Some happen at search time: the answer is not in the documents at all, or the right document does not rank high enough to be used. Some happen when the system picks what the model reads: the right document is found but does not make it into the context. And some happen at writing time: the answer is in front of the model, but it fails to pull it out, typically when there is "too much noise or contradicting information". The list also includes the wrong format, the wrong level of detail, and incomplete answers.

The missing content case worries me most. When the answer is not in the documents, the report says the system "could be fooled into giving a response" instead of saying it does not know. That ties into a wider problem I have written about: AI tests should reward "I don't know" instead of a confident guess. Stuffing more documents into the prompt is not a fix either, for the same reason a bigger context window is not better memory.

The report's two main takeaways should sober up anyone buying a RAG product: "validation of a RAG system is only feasible during operation", and the robustness of a RAG system "evolves rather than designed in at the start". In plain terms, you find out how well it works by watching it work.

A citation is not proof

Citations are the main reason people trust RAG answers more than plain chatbot answers. I think that trust is often misplaced. A citation is the system saying "this source supports me". That statement can be wrong, and the legal study shows it can be wrong in a way that looks right.

This is a design choice as much as a technical limit. A tool can make checking easy: link to the exact passage, show what it searched, and say plainly when it found nothing. Or it can make checking hard and lean on the look of authority. It is the same trap as a chatbot that always agrees with you: the product feels helpful while it quietly moves the work of checking onto you.

What I would ask before trusting a RAG tool

I would ignore the RAG label and ask three things.

  • How good is the search? Can I see which documents it used? Does it tell me when the answer is not in its sources, or does it answer anyway?
  • How easy is it to check a citation? Does each claim link to the exact passage that supports it, or to a long document I have to read in full?
  • Has anyone tested it on my kind of question? The legal study found large differences between tools that all use retrieval. Only a test on real questions shows which kind of tool you have.

The Stanford and Yale authors end with advice that holds for every field, not just law: "Users of these tools must continue to verify that key propositions are accurately supported by citations." RAG makes AI answers better. It does not make checking them optional.

Sources

OpinionRAGLLM