What Makes an AI Eval Defensible to Auditors?

3 min read ▶ Listen to this post

An AI eval is defensible when it can survive someone else trying to poke holes in it — a customer's technical team, a compliance reviewer, your own board — not just when it produces a good-looking number. That means showing the work behind the score: what was tested, how it was scored, what failed, and why the test set and the scoring method can be trusted in the first place.

What "defensible" actually requires

• A described test set, not just a score. Where the test cases came from, how many there are, and whether they overlap with anything the model could have seen during training or fine-tuning. A number with no description of its source isn't something anyone outside your team can evaluate.
• A scoring method checked against a human baseline. If an LLM or a rubric is doing the scoring, someone needs to have checked that its judgments agree with careful human judgment — and checked it in the places that matter most, not just on average across easy cases.
• Failure cases included, not filtered out. An eval report that shows only where the system performed well reads, to anyone experienced in reviewing these, as one that's hiding the rest. Reporting the failures and explaining what you did about them is what makes the good results credible.
• A stated scope. What the eval does and doesn't cover — which use cases, which user populations, which kinds of inputs were and weren't tested. An eval that implies it covers everything usually hasn't actually tested most of it.

Where most evals fall apart under questioning

The most common failure is a single aggregate accuracy number with nothing behind it — no breakdown by category, no subgroup analysis, no description of what a wrong answer actually looked like. The second most common is a test set that was also used, in some form, to tune the system, which quietly turns a held-out evaluation into a training metric wearing an evaluation's clothes. The third is treating one strong correlation with human judgment as proof the system is safe to deploy everywhere, when a correlation computed across easy cases can hide serious disagreement in the small number of cases that actually carry the risk.

What to build before anyone asks

Keep a written record, from the start, of where every test case came from, how scoring decisions were made, and which categories or subgroups were checked separately rather than only in aggregate. Re-run the eval whenever the system changes, and keep the old results — a customer or auditor asking "has this changed since the version you evaluated" deserves a real answer, not a guess. None of this needs to be elaborate. It needs to exist, be dated, and be specific enough that someone outside your team could look at it and understand exactly what was and wasn't tested.

If you're building an eval that needs to hold up to outside scrutiny — a customer's due diligence, a regulator, or your own leadership — this is the kind of evaluation work we help teams build.

Related reading

• How to Evaluate an LLM Beyond Average Accuracy — the deeper case for checking subgroup and stability diagnostics instead of a single aggregate score, one of the habits a defensible eval depends on.
• The 1979 IBM Warning That AI Still Hasn't Answered — why accountability, not just accuracy, is the standard a defensible AI system actually has to meet.

One email whenever we publish something new. No spam, unsubscribe anytime.