AI Hallucinations: The Specific Reason Models Confidently Make Things Up

A Concrete Example of the Problem

In their 2025 paper "Why Language Models Hallucinate," OpenAI researchers Adam Tauman Kalai, Ofir Nachum, Santosh Vempala, and Edwin Zhang describe testing a widely used chatbot with a simple factual question: what was the title of one author's actual PhD dissertation? The model confidently produced three different answers across three attempts — none of them correct. Asked for that same person's birthday, it again gave three different, entirely wrong dates, each stated with full apparent confidence.

This kind of confidently stated falsehood is what's commonly called an AI "hallucination" — plausible-sounding, fluently written, completely wrong information, presented with no indication of uncertainty. It's one of the most persistent, widely reported problems across essentially every major AI language model, and it hasn't gone away as models have improved in other respects.

Why This Happens at the Most Basic Level

The researchers' first key finding is a statistical one: language models are trained to predict the most likely next word in a sequence of text, based on patterns learned from enormous amounts of training data. When a model is asked something for which the correct answer wasn't clearly and consistently represented in its training data — an obscure fact, a very specific detail, information that simply isn't well documented anywhere the model learned from — the model doesn't have a reliable way to distinguish "I actually don't know this" from "I know this." It just predicts the most statistically plausible-sounding continuation of text, which often looks exactly like a confident, correct answer, whether or not it actually is one.

The researchers frame this mathematically as a version of a well-understood problem in machine learning: generating a correct, specific answer to a question is fundamentally harder than simply judging whether a given, already-written answer is true or false. Their formal analysis shows that a model's error rate when generating new answers must be at least roughly twice as high as its error rate when just classifying existing statements as true or false. In other words, some level of hallucination isn't a random flaw that better engineering can simply eliminate — it's a near-mathematical consequence of the basic task these models are trained to perform.

The Bigger Problem: How Models Get Graded

The researchers argue that a second factor makes the situation meaningfully worse, and it has nothing to do with the model's core architecture at all — it's about the tests, called benchmarks, used to measure and reward good AI model performance during training and development.

Most standard AI evaluation benchmarks score answers using simple binary correct-or-incorrect grading — a model gets full credit for a right answer, zero credit for a wrong one, and typically zero credit for honestly saying "I'm not sure" as well. Under that specific grading scheme, a model that always guesses confidently, even when it doesn't actually know the answer, will score better on average than an equally accurate model that sometimes truthfully admits uncertainty — purely because guessing occasionally gets lucky and produces a right answer by chance, while honest uncertainty never scores any points at all. The researchers describe this as similar to a student taking a multiple-choice exam: guessing on questions you don't know the answer to, rather than leaving them blank, is usually the better test-taking strategy, even though it has nothing to do with actual knowledge.

Because so many widely used, high-profile benchmarks are built this way, models that are deliberately or incidentally optimized to perform well on them are being systematically pushed toward confident guessing and away from honestly signaling uncertainty — even in cases where the model's internal processing genuinely reflects low confidence in its own answer.

Why Bigger, Newer Models Haven't Solved This

OpenAI's own research notes that newer models, including GPT-5, do show meaningfully reduced hallucination rates compared to earlier systems, particularly on tasks involving more deliberate step-by-step reasoning — but hallucinations haven't disappeared, and the researchers argue they won't disappear through model scale and improvement alone as long as the underlying evaluation and grading systems continue to reward confident guessing over honest uncertainty. Some real-world questions simply don't have a determinable, verifiable answer available anywhere in a model's training data or accessible tools — due to missing information, genuine ambiguity, or limits on what any model could reasonably know — and a grading system built entirely around raw accuracy will always create pressure to guess anyway rather than say so.

The Proposed Fix

The researchers' recommended solution isn't a purely technical patch to model architecture — it's a change to how the entire field measures and rewards good performance. Specifically, they argue benchmarks should be redesigned to penalize confident wrong answers more heavily than they currently do, while giving explicit partial credit for appropriately expressed uncertainty, rather than continuing to treat "I don't know" as functionally equivalent to a wrong answer. Because so many different organizations and researchers rely on the same small set of widely used, influential benchmarks to evaluate progress, the researchers describe this as a "socio-technical" problem — meaning it can't be solved by any single AI lab quietly retraining their own model. It requires a broader, coordinated change in how the entire field agrees to measure success in the first place.