Blog topic

AI Evaluation

For AI startups, product teams, edtech and healthcare organizations, and anyone who needs to know whether an AI system actually does what its numbers claim. These articles apply measurement and statistics to AI quality: validity, reliability, benchmark contamination, and comparing AI judgments with human ones.

Browse topics

All articles Study Design Program Evaluation AI Evaluation Statistical Consulting Data Literacy

Featured

{{ p.readingTime }} min read

AI Evaluation

The Test GPT-3 Had Already Studied For

4 min read

In 2020, OpenAI checked whether GPT-3's benchmarks had leaked into its training data. For most, it didn't matter — for two, it clearly did.

Read post →

AI Evaluation

The 1979 IBM Warning That AI Still Hasn't Answered

4 min read

A decades-old training-manual slide argued a computer can never be held accountable, so it must never make a management decision. Two 2025-2026 studies now show, in quantified detail, exactly why that gap hasn't closed.

Read post →

AI Evaluation

The AI Raters That Scored Like Humans and Judged Nothing Like Them

4 min read

Nine LLMs rating hate speech correlated with human judgments at 0.911 — a number that would pass almost any validation check. Decomposed into its parts, the two scales disagreed on severity, item calibration, question order, and who the comment was about.

Read post →

AI Evaluation

How to Evaluate an LLM Beyond Average Accuracy

2 min read

A single accuracy number can hide the exact failures that matter most — subgroup gaps, instability across runs, and confident wrong answers. Here's what a more defensible evaluation actually measures.

Read post →

AI Evaluation

The Essay That Said Nothing and Scored a Perfect 6

4 min read

An automated essay-grading engine gave top marks to student essays that made no sense at all, as long as they used long sentences and fancy words. A real case study in Goodhart's Law, and why optimizing a proxy metric isn't the same as measuring the thing you actually care about.

Read post →

AI Evaluation

The Wolf Detector That Never Looked at the Wolf

4 min read

A machine-learning model hit strong accuracy telling wolves from huskies — until researchers checked what it was actually looking at. A real case study in shortcut learning, and why a good accuracy number can hide exactly how a model is cheating.

Read post →

{{ p.readingTime }} min read

Read post →

Evaluating an AI system?

We bring measurement and statistical rigor to AI and LLM evaluation — beyond a single accuracy number.

AI evaluation services Book a free consult

← Back to all articles

Get new posts by email

One email whenever we publish something new. No spam, unsubscribe anytime.

Check your inbox — click the confirmation link to finish subscribing.

AI Evaluation

Browse topics

Featured

{{ p.title }}

{{ listHeading }}

{{ p.title }}