Blog topic

AI Evaluation

For AI startups, product teams, edtech and healthcare organizations, and anyone who needs to know whether an AI system actually does what its numbers claim. These articles apply measurement and statistics to AI quality: validity, reliability, benchmark contamination, and comparing AI judgments with human ones.

Series · 2 stories

The Judgment Gap

AI made answers cheap. It did not make judgment cheap. A series on why experience still matters, and how to use AI without handing it your judgment.

Series · 2 posts

AI Eval Watch

A weekly look at new research and vendor claims about evaluating AI, with a statistician's view of what each one means for how you test and compare models.

Browse topics

All articles Statistical Analysis Survey Design & Measurement Study Design Program Evaluation AI Evaluation Statistical Consulting Data Literacy

Featured

{{ p.readingTime }} min read

AI Evaluation

AI Eval Watch, Oct 21–27: Label Budgets, Missed Bugs and Benchmark Shortcuts

6 min read

This week's AI-evaluation research asked what a score really measures: cheaper accuracy audits, coding agents judged against a narrow reference, fact-checking scores that hide gaps, and a ranking metric that masked a failing defense.

Read post →

AI Evaluation

AI Eval Watch, Oct 14–20: Judges, Benchmark Noise and a Self-Graded Leaderboard

5 min read

This week's AI-evaluation research was about trusting the grader and the score: a self-debiasing LLM judge, screening judges with no ground truth, noisy agent benchmarks, and a vendor-graded search leaderboard.

Read post →

Statistical AnalysisAI Evaluation

When AI-Written Code Runs, What Has It Proved?

5 min read

A clean run proves AI-written analysis code is valid, not that it estimates what you meant. Four checks that produce real validation evidence.

Read post →

Program EvaluationAI Evaluation

An Active AI Tutor Isn't Yet an Effective One

4 min read

Logins and minutes show an AI tutor was used. Whether students learned needs an outcome they complete without it, and a comparison chosen in advance.

Read post →

Study DesignAI EvaluationSurvey Design & Measurement

How Do You Measure Inter-Rater Reliability?

3 min read

Percent agreement and correlation can look excellent when raters barely agree. How to choose between Cohen's kappa, Krippendorff's alpha, and the ICC.

Read post →

Data LiteracyAI Evaluation

The 27 Programs That Failed Together

5 min read

Twenty-seven programmers wrote the same program separately, so their bugs should have been independent. On 1,255 test cases they failed together.

Read post →

AI Evaluation

AI Can Think for You, But It Can't Understand for You

5 min read

Students with an unguarded AI tutor practiced better and then scored 17% worse on the exam once it was gone. AI can supply the reasoning. Owning it, and being able to defend it, is still your job.

Read post →

AI Evaluation

The Bitter Lesson Has a Second Half

4 min read

Chess, Go, and now formal mathematics all fell to search and learning. Each had a referee that could say, without argument, who was right. Most AI products have no such referee, and that is where the statistics come in.

Read post →

AI Evaluation

The Test Questions That Were Failing the AI

7 min read

A grade-school math benchmark marked "$7.00" wrong and "$7" right. Borrowing item analysis from psychometrics, Stanford researchers flagged broken questions in nine widely used AI benchmarks.

Read post →

AI Evaluation

The AI Users Who Scored 13 and Guessed 17

6 min read

Given ChatGPT and 20 LSAT logic problems, people scored about 13 and were sure they had scored about 17. The gap between having an answer and knowing it is where AI makes judgment harder, not easier.

Read post →

AI Evaluation

How Many Test Items Does an AI Eval Need?

5 min read

An accuracy score from 200 items can be off by five points either way, and comparing two models takes far more items than most teams expect. How to size an eval for the question it has to answer.

Read post →

AI Evaluation

How to Tell If One LLM Is Really Better Than Another

5 min read

A new model or prompt scores three points higher on your eval. Before you switch, check whether the gap is bigger than luck: a worked example, the paired test to use, and three ways a false win gets in.

Read post →

AI Evaluation

How to Vet an AI Vendor's Accuracy Claims

3 min read

A vendor's "95% accuracy" claim is only as good as the test set, scoring method, and baseline behind it. Ask for those before you believe it.

Read post →

AI Evaluation

What Makes an AI Eval Defensible to Auditors?

3 min read

A defensible AI eval documents its test data, scoring method, and failure cases so it holds up to outside scrutiny, not just a good accuracy score.

Read post →

AI Evaluation

The Test GPT-3 Had Already Studied For

4 min read

In 2020, OpenAI checked whether GPT-3's benchmarks had leaked into its training data. For most, it didn't matter — for two, it clearly did.

Read post →

AI Evaluation

The 1979 IBM Warning That AI Still Hasn't Answered

4 min read

A decades-old training-manual slide argued a computer can never be held accountable, so it must never make a management decision. Two 2025-2026 studies now show, in quantified detail, exactly why that gap hasn't closed.

Read post →

AI EvaluationSurvey Design & Measurement

The AI Raters That Scored Like Humans and Judged Nothing Like Them

4 min read

Nine LLMs rating hate speech correlated with human judgments at 0.911 — a number that would pass almost any validation check. Decomposed into its parts, the two scales disagreed on severity, item calibration, question order, and who the comment was about.

Read post →

AI Evaluation

How to Evaluate an LLM Beyond Average Accuracy

2 min read

A single accuracy number can hide the exact failures that matter most — subgroup gaps, instability across runs, and confident wrong answers. Here's what a more defensible evaluation actually measures.

Read post →

AI Evaluation

The Essay That Said Nothing and Scored a Perfect 6

4 min read

An automated essay-grading engine gave top marks to student essays that made no sense at all, as long as they used long sentences and fancy words. A real case study in Goodhart's Law, and why optimizing a proxy metric isn't the same as measuring the thing you actually care about.

Read post →

AI Evaluation

The Wolf Detector That Never Looked at the Wolf

4 min read

A machine-learning model hit strong accuracy telling wolves from huskies — until researchers checked what it was actually looking at. A real case study in shortcut learning, and why a good accuracy number can hide exactly how a model is cheating.