This is an opinion piece. The facts come from the linked sources; the argument is mine.
When a chatbot states something false with total confidence, it is easy to treat that as a flaw in the software. I think a large part of the flaw sits upstream, in the tests we use to rank these systems. Most leading benchmarks give zero points for admitting ignorance and full points for a lucky guess, so models learn to guess. Change the scoring and you change the behavior. Readers can change what they reward as well, by judging AI tools on how often they are wrong and how honestly they flag uncertainty, not only on how often they are right.
Why chatbots guess instead of saying "I don't know"
In September 2025, OpenAI published a plain-English account of this problem. Its blog post defines hallucinations as "plausible but false statements generated by language models." That is the familiar experience of an answer that reads smoothly and is simply untrue.
The central claim of that post is uncomfortable for the field. It rests on a research paper that "argues that language models hallucinate because standard training and evaluation procedures reward guessing over acknowledging uncertainty."
Think about multiple-choice tests in school. If you do not know the answer, a guess might get lucky, and "Leaving it blank guarantees a zero." So you guess. The blog uses a birthday example. If a model asked for a birthday it does not know guesses "September 10," it has "a 1-in-365 chance of being right", while admitting ignorance earns nothing. Under that scoring, guessing is the rational move even though it is nearly always wrong.
The paper, by Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang of OpenAI and Georgia Tech, makes the same point in its abstract: models are "optimized to be good test-takers, and guessing when uncertain improves test performance." The authors examined 10 popular benchmarks, including GPQA, MMLU-Pro, SWE-bench and HLE. Nine of them grade answers as simply right or wrong and give no credit for admitting uncertainty. Only one, WildBench, offers partial credit, and the authors note that even its rubric suggests an admission of uncertainty may score lower than a fairly rated answer that contains a hallucination.
One lucky guess looks trivial. The damage is statistical. In the blog's words, "Over thousands of test questions, the guessing model ends up looking better on scoreboards than a careful model that admits uncertainty." These tests do not merely fail to catch bluffing. They pay for it.
What the numbers show
OpenAI's blog cites SimpleQA results from its GPT-5 System Card comparing two of its own models. The newer gpt-5-thinking-mini abstained 52 percent of the time, answered 22 percent correctly and got 26 percent wrong. The older OpenAI o4-mini almost never abstained, at just 1 percent. It answered 24 percent correctly and got 75 percent wrong.
Do the arithmetic with me. The older model was wrong almost 2.9 times as often, 75 percent against 26 percent, while scoring only 2 percentage points higher on accuracy, 24 against 22. OpenAI acknowledges the older model "performs slightly better" on accuracy, while its error rate "is significantly higher."
Ranked on accuracy alone, the riskier model wins. Ranked on how much harm it does to a user who trusts it, the careful model wins by a wide margin. OpenAI itself says "accuracy-only scoreboards dominate leaderboards and model cards," which is one more reason I have argued that a leaderboard rank tells you less about an AI model than you think.
OpenAI also pushes back on the idea that only bigger models can avoid hallucinations. It says "It can be easier for a small model to know its limits."
The fix is a scoring change
The remedy proposed in the paper is plain. Rather than building yet another hallucination test, the authors favor "modifying the scoring of existing benchmarks that are misaligned but dominate leaderboards, rather than introducing additional hallucination evaluations."
The mechanics they describe involve a confidence target written into the test instructions. In one example they give, the model should answer only if it is more than 75 percent confident. A wrong answer costs 2 points, a right answer earns 1 point, and an admission of uncertainty scores 0. The paper's general rule is a penalty of t divided by (1 minus t) points for a target of t. By that formula, a 75 percent target works out to a 3-point penalty, and the 2 points the paper lists match a target of about 67 percent. The exact numbers matter less than the direction. Under a rule like that, guessing a random birthday stops being a smart bet and becomes a losing one, because a confident error costs more than silence.
OpenAI's blog endorses the same direction: "Penalize confident errors more than you penalize uncertainty, and give partial credit for appropriate expressions of uncertainty." It also points out that none of this is exotic, since "Some standardized tests have long used versions of negative marking for wrong answers or partial credit for leaving questions blank to discourage blind guessing."
Why would a scoring change alter behavior and not just measurement? Because, in OpenAI's words, accuracy-only scoreboards end up "motivating developers to build models that guess rather than hold back." The blog puts it bluntly: "If the main scoreboards keep rewarding lucky guesses, models will keep learning to guess." Reverse the payoff and the incentive reverses with it.
What this does not solve
The authors themselves flag limits, and I take those limits seriously.
First, better tools do not remove the problem. The authors write that "Search (and reasoning) are not panaceas." Their point is that right-or-wrong grading still rewards guessing whenever a search fails to turn up a confident answer.
Second, the authors call the simple split of answers into right, wrong and I-don't-know "A false trichotomy" because their framework does not separate errors of different sizes. A small slip and an invented fact are not the same failure, yet a coarse score treats them alike. Scoring reform blunts the incentive to guess; it does not measure how much harm an error causes.
Third, perfection is not coming. OpenAI says accuracy "will never reach 100%" because "some real-world questions are inherently unanswerable." Even an honest system will sometimes be wrong. The difference is that it tells you when it is unsure.
Fourth, the paper's own abstract hedges its conclusion, saying the change "may steer the field toward more trustworthy AI systems." Scoring is a lever, not a cure. My own worry, and this is opinion, is the opposite failure mode: a system that abstains unless it feels highly confident could decline so many ordinary questions that it stops being useful. Helpful and careful are in tension, and a scoring rule alone does not resolve that tension.
What it means for you
You do not run benchmarks, but you do pick tools, and you do decide what counts as a good answer.
Judge tools by how often they are wrong, not only by how often they are right. When a company shares results, look for the error rate and how often the system declines to answer. As the SimpleQA comparison shows, two similar accuracy numbers can hide very different reliability.
Value a tool that tells you it is unsure. A chatbot that always has an answer is not more capable. It may simply be more willing to guess. I have argued that an AI chatbot that always agrees with you is a design choice, and the same goes for a chatbot that never admits doubt. Makers choose that behavior, and you can choose different products.
Ask for uncertainty directly. Tell the chatbot to say when it is not confident and to separate what it knows from what it is inferring. That is no substitute for fixing the scoring, but it changes the conversation you have with the tool today.
Check sources for anything that matters. Ask where an answer came from and verify it yourself. Our explainer on why AI chatbots make things up and how to catch them walks through how to spot a fabrication.
The bottom line
Chatbots bluff partly because bluffing pays. The tests that rank them award full credit for a lucky guess and zero for honesty, and nine of the 10 benchmarks the authors examined work exactly that way. The proposed fix is modest and concrete: penalize confident errors, give partial credit for admitted uncertainty, and write confidence targets into the tests themselves. Until the scoreboards change, you can apply the same standard by hand. Ask any AI tool two questions: how often is it wrong, and does it tell you when it does not know? A system that can say it does not know is not weak. It is honest, and honesty is something you can check.
