Every production LLM still makes things up. Not because the engineers were careless, and not because the architecture is cursed, but because guessing is what the training objective rewards. This is the explainer version of that story: what a hallucination actually is, why three different stages of the pipeline produce them, and what current research says about detecting and reducing them.
What counts as a hallucination
In natural language generation research, the standard taxonomy comes from a 2023 ACM Computing Surveys survey by Ji et al.. It splits hallucinations into two kinds. Intrinsic hallucinations contradict the provided source material: the model is summarizing a document and states something the document denies. Extrinsic hallucinations go beyond the source: the model adds claims that sound plausible but cannot be verified against anything given. The survey covers all NLG tasks; for open-ended chat models, extrinsic hallucination against world knowledge dominates the conversation.
OpenAI's 2025 paper "Why Language Models Hallucinate" (Kalai, Nachum, Vempala, Zhang) narrows further to the most stubborn subtype: overconfident, plausible falsehoods stated as fact. The Oxford team that published in Nature calls this confabulation: generations that are both incorrect and arbitrary, in the sense that they change with the sampling seed. A model that is confidently wrong about one fact in ten is hallucinating. A model that gives a different wrong answer every time you re-roll is confabulating.
The authors' opening example is the right tone for the problem. Asked for the title of one of the authors' PhD dissertation, three popular models produced three different answers, with three different years and three different universities. None were correct. The same paper reports a state-of-the-art open model giving three wrong dates when asked a person's birthday, despite being told to answer only if it knew.
Cause one: pretraining is a guessing machine
The intuitive story says hallucination is a data quality problem: the model memorized garbage from the internet. The Kalai et al. analysis is more uncomfortable.

They show that even if the training corpus were error-free, the statistical objectives used to train language models would still produce errors.
The reduction is simple. Next-token prediction over a huge fact space is effectively a binary classification task per fact: valid statement versus error. Rare, singleton facts (one person's birthday, one dissertation title) are statistically indistinguishable from the plausible variants the model can generate around them. A model that has seen a fact exactly once cannot separate "this is the fact" from "this is one of thousands of strings that look like the fact." Under any loss that scores full sentences by likelihood, errors in that regime are not a bug you can train away; they are the expected outcome of the estimation problem. As the paper puts it, hallucinations originate simply as errors in binary classification.
Cause two: evaluations grade test-takers, not honest students
If pretraining creates the errors, post-training should fix them. Partially, it does. But Kalai et al. argue the incentive structure of the evaluation ecosystem keeps hallucination in business. Almost every headline benchmark scores binary accuracy: right or wrong, no partial credit for saying "I don't know." A model maximizing benchmark scores therefore maximizes guessing, because an uncertain guess can score points while abstention never does.
The numbers are stark. The Humanity's Last Exam leaderboard, one of the most-cited hard benchmarks, scores binary accuracy with no credit for abstention, and at the time of the paper every reported model sat below 30% accuracy. Most models also showed calibration error above 70%, meaning their stated confidence poorly tracks their actual accuracy. The authors' proposed fix is socio-technical rather than model-side: change the scoring of the benchmarks that dominate leaderboards so that calibrated abstention is rewarded, instead of adding yet another isolated hallucination benchmark nobody optimizes against.
Cause three: inside the model, refusal is the default that gets overridden
Anthropic's 2025 circuit-tracing work looked inside Claude 3.5 Haiku and found a counterintuitive mechanism. Default behavior is not to answer. A circuit that is "on" by default makes the model state it has insufficient information. When a question touches something the model knows well, a "known entities" feature activates and inhibits that refusal circuit, letting the answer through.
Hallucinations then look like misfires of the recognition system. When Claude recognizes a name but knows nothing else about the person, the "known entity" feature can still fire and suppress the honest refusal. Once the refusal is inhibited, the model commits to answering and confabulates. Anthropic demonstrated causality both ways: artificially activating the "known answer" features caused the model to hallucinate consistently that an unknown person plays chess, and the same mechanism, running naturally, produces ordinary hallucinations. The practical lesson: a model making up a fact is not always a model that "thinks" the lie; sometimes it is a model that was structurally denied the chance to say it did not know.
Detection: measure uncertainty over meanings, not words
If models cannot always be trusted to flag their own guesses, can an external detector do it?

Farquhar, Kossen, Kuhn and Gal published semantic entropy in Nature in June 2024. The method samples the same question many times, clusters the answers by meaning rather than by surface wording (so "Paris", "It's Paris" and "The capital of France is Paris" land in one cluster), and computes entropy over the clusters. Low semantic entropy means the model converges on one meaning; high entropy across mutually exclusive meanings flags likely confabulation. The paper reports the method works across datasets and tasks without task-specific training data, and its detection of confabulation generalized to new question types.
The cost is the caveat. A follow-up analysis notes the Nature protocol uses roughly 50 samples per question conditioned on a 20-shot prompt, on the order of 100x the compute of a single generation. Oxford's own Semantic Entropy Probes paper proposes cheaper approximations by reading uncertainty signals off internal activations. So semantic entropy is currently a quality-control tool for high-stakes offline evaluation, not something you run on every chat reply at production traffic.
What actually reduces it
Grounding the generation in retrieved evidence addresses extrinsic hallucinations at the source: Lewis et al. 2020 introduced retrieval-augmented generation precisely to let models pull verifiable text at generation time and cite it. It changes the failure mode from invention to misquotation, which is cheaper to audit. On the alignment side, models trained to abstain do refuse more (Anthropic's circuit work describes this as successful though imperfect anti-hallucination training), and earlier work had already shown models carry some internal signal of what they know (Kadavath et al. 2022). What none of this does is eliminate the statistical floor from pretraining. For long-tail facts, guessing remains a property of the objective, not of a particular model.
The honest summary for anyone deploying these systems: treat fluency as zero evidence of correctness, prefer answers that cite verifiable sources, assume low performance on rare facts, and reward abstention in your own evals if you want it in production behavior.
Sources
- Kalai, Nachum, Vempala, Zhang, Why Language Models Hallucinate, OpenAI / Georgia Tech, September 2025. Full PDF.
- Ji et al., Survey of Hallucination in Natural Language Generation, ACM Computing Surveys, 2023.
- Farquhar, Kossen, Kuhn, Gal, Detecting hallucinations in large language models using semantic entropy, Nature 630, June 2024.
- Anthropic, Tracing the thoughts of a large language model, May 2025.
- Semantic Entropy Probes, Kossen, Farquhar et al., 2024.
- Layer-wise information deficiency for hallucination detection, 2024/2025 (semantic entropy compute-cost baseline).
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020.
- Kadavath et al., Language Models (Mostly) Know What They Know, 2022.
