Most eval sets are sized by what was easy to collect: a few hundred prompts someone wrote, a benchmark's test split, whatever the labeling budget covered. The right size depends on what the eval has to answer. "How accurate is this model?" and "Is the new version better than the old one?" are different questions, and the second usually needs several times more items.
An accuracy score measured on a sample of items is an estimate, and its margin of error shrinks only with the square root of the number of items. For a model that gets about 80% right, the 95% margin of error is roughly:
To plan, work backwards: pick the margin you can live with and solve for the number of items, n = 1.96² × p(1 − p) ÷ margin², where p is the accuracy you expect. For accuracy near 80%, a ±5-point margin takes 246 items, ±3 points takes 683, and ±2 points takes 1,537. Near 90% you need fewer (139, 385 and 865), because scores close to the ceiling vary less. If you have no idea what to expect, plan for 50%, the worst case: 385, 1,068 and 2,401 items.
The practical consequence: on a 200-item eval, a reported 80% is consistent with a true accuracy anywhere from about 74% to 85%. Reporting it as 80.5% implies a precision the eval doesn't have.
Comparing two models is where most evals come up short, because the comparison depends on a different number: how often the two models disagree. When both models answer the same items, every item they both get right, or both get wrong, says nothing about which is better. The comparison rests entirely on the items where one was right and the other wrong. (That's why the comparison should be paired: run both models on the same items and compare them item by item. Our guide to comparing two models walks through the test.)
So the number of items depends on two inputs: the smallest accuracy gap you need to detect, and the share of items on which the models will disagree. For an 80% chance of detecting the gap at the usual 5% significance level (Connor, 1987):
We checked each of these by simulation, using the same test as our free model comparison calculator. Every one came out at 80% power, give or take a point.
Two patterns stand out. Small gaps are expensive: halving the gap you want to detect roughly quadruples the items. And more disagreement means more items for the same gap, not fewer: when two models disagree often, a small net difference is harder to tell apart from chance.
To estimate the disagreement rate, run both models on a pilot of a few hundred items and count. The accuracy gap on a pilot is too noisy to plan with, but the disagreement rate is estimated well enough; to be safe, plan with the upper end of its likely range.
Pairing is also what keeps these numbers manageable. If the two models were scored on different item sets, detecting 80% versus 83% would take about 2,600 items per model. Scoring both on the same items is the cheapest way to make an eval more sensitive.
All of these numbers assume every item is a separate, independent test. Eval sets often aren't: five questions about the same document, several paraphrases of one prompt, turns from one conversation. Items in a group tend to succeed or fail together, so they carry less information than the same number of independent items.
The standard correction is the design effect, 1 + (m − 1) × ICC, where m is the number of items per group and the ICC (intraclass correlation) measures how alike items in the same group are. Take 500 items built from 100 documents with 5 questions each, and a modest ICC of 0.2: the design effect is 1.8, so the eval carries about as much information as 280 independent items, and the ±3.5-point margin grows to ±4.7. Either collect more groups (more documents beats more questions per document) or report clustered standard errors, as Miller (2024) recommends for language model evals.
If you're sizing an eval that has to hold up to a customer, an auditor or your own leadership, this is the kind of evaluation design we help AI teams with.
One email whenever we publish something new. No spam, unsubscribe anytime.
{{ subscribeErrorMsg }}