← Back to blog
AI Evaluation

Same Model, Different Hallucination Rate: A Construct-Validity Lesson

By Dr. Nedim Yel, Co-Founder & Lead Statistician · Updated
3 min read

"What is the hallucination rate of this model?" sounds like a question with one answer. It does not have one. Two well-known public measures show why.

The first is Vectara's Hallucination Leaderboard. It gives each model a set of short documents, mostly news articles from the CNN/Daily Mail corpus, and asks for a summary. A separate scoring model, Vectara's HHEM, then rates each summary against its source on a scale from 0 to 1, and a score below 0.5 counts as a hallucination. The reported rate is the share of summaries flagged. Refusals to summarize are not counted against the model, and an answer rate is reported alongside. Vectara states that the leaderboard measures hallucination within summarization specifically, not overall summary quality or general question answering.

The second is the AA-Omniscience benchmark from Artificial Analysis, covering 6,000 questions across 42 topics in six domains such as business, health, law, software engineering and science. It asks questions that require the model to know facts, and it rewards correct answers, penalizes wrong ones, and does not penalize declining to answer. Its hallucination measure concerns how often a model answers incorrectly when it should have admitted it did not know.

Same word, different question

The first benchmark asks: when a model is handed the source text, does it stay faithful to it? The second asks: when a model must rely on what it learned in training, does it make things up or say it does not know? A model can be good at one and weak at the other. A "hallucination rate" from the first says little about the second, and the reverse is also true. Even the two definitions of the second measure differ across sites that report it: one counts wrong answers among cases where the model should have abstained, another counts them among all answers that were not correct. Those are different denominators, so the percentages are not interchangeable.

There is also a scoring question. The first benchmark uses another AI model to judge, which introduces that judge's own error rate. The second depends on how questions were chosen and graded. Neither is wrong. Each produces a number that is only meaningful with its definition attached.

Where else this shows up

  • AI evaluation. "Accuracy," "safety" and "helpfulness" scores from different tests can share a label and share almost nothing else. Compare numbers only within one test and one version.
  • Program evaluation. "Completion rate" can mean finished the first session, finished the full curriculum, or finished within a year. The same program can look very different under each.
  • Academic research. A construct such as "anxiety" or "reasoning" measured by different instruments can yield incompatible results while carrying the same name.

What to ask for instead

When someone quotes a rate, ask what is in the numerator and the denominator, what task the model was doing, who or what did the scoring, and which version of the test and the model it applies to. Then ask whether that task resembles yours. A model that summarizes documents faithfully may still be a poor source of facts, and the reverse.

If you are comparing AI tools for a specific use, or need an evaluation built around your own tasks rather than someone else's metric, send us what you are trying to decide. Working out what to measure is usually a short conversation.

Related reading

Sources and further reading

Get new posts by email

One email whenever we publish something new. No spam, unsubscribe anytime.

Check your inbox — click the confirmation link to finish subscribing.

{{ subscribeErrorMsg }}