How to Evaluate an LLM Beyond Average Accuracy

"Our model is 92% accurate" is a headline, not an evaluation. A single average obscures exactly the information a customer, auditor, or your own engineering team needs: where does it fail, how consistently, and how confidently wrong is it when it fails? An average accuracy figure can be technically true and still misrepresent how the system will actually behave in production.

What an average hides

• Subgroup performance. A model can hit 92% overall while performing dramatically worse on a specific input type, language, or user population — the exact cases most likely to surface once you're at scale with real users.
• Run-to-run variance. LLM outputs vary across prompts, sampling temperature, and even identical inputs. A single accuracy run tells you nothing about how stable that number is — report it with a confidence interval or a distribution across repeated runs, not a point estimate.
• Calibration. A model that's wrong 20% of the time but always sounds equally confident is more dangerous in practice than one that's wrong 25% of the time but signals uncertainty when it's guessing. Calibration — whether stated confidence matches actual correctness — is a separate axis from accuracy.
• Failure severity, not just failure rate. Not all errors cost the same. A defensible evaluation weights errors by consequence, not just by count.

What a more defensible evaluation looks like

Stratify results by the subgroups and input types that matter for your use case, not just an overall average. Report a confidence interval or variance estimate, not a single number, especially when comparing two model versions — a 1-point difference in average accuracy is often not statistically distinguishable from noise. Pair automated metrics with structured human evaluation on a representative sample, since automated metrics (like exact-match or embedding similarity) often miss the specific failure modes that matter to your product. And when comparing model versions or vendors, use a paired statistical test on the same evaluation set rather than comparing two separately-reported averages.

Why this matters before a customer asks

The organizations getting burned right now aren't the ones with bad models — they're the ones whose evaluation couldn't answer a customer's or auditor's specific question about where and how the system fails. That gap is usually a statistics problem, not a machine-learning problem.

If you're building evaluation metrics for an AI product and need them to hold up under real scrutiny, this is squarely what we do.

One email whenever we publish something new. No spam, unsubscribe anytime.