A new model version, or a rewritten system prompt, scores 81% on your eval where the old one scored 78%. Three points is a meaningful improvement if it's real. The question is whether it is: every eval score has sampling error, and the gap between two noisy scores can easily be noise too.
Suppose both versions answered the same 300 eval items. The new one was right on 243 (81.0%) and the old one on 234 (78.0%). Now look item by item:
The whole comparison rests on that 27-to-18 split. If the two versions were equally good, each disagreement would be a coin flip, and a 27-to-18 split is well within what coin flips produce (McNemar mid-p = .18). The 95% confidence interval for the new version's true advantage runs from −1.4 to +7.4 points. The eval is consistent with a real 7-point gain and with a small loss. It can't tell these versions apart.
Now the same 3-point gap on 1,000 items, with a 90-to-60 split of the disagreements: mid-p = .014, and the interval runs from +0.6 to +5.4 points. That eval does show the new version is better, although the gain could be as small as about half a point.
You can reproduce both results, or run your own counts, with our free model comparison calculator. It also gives you R and Python code for the same analysis.
Because both versions answered the same items, their scores aren't independent: an easy item tends to be easy for both. A test that treats the two accuracies as separate samples, such as the usual two-proportion z-test, ignores that. On the two examples above it gives p = .36 and p = .097, so on the 1,000-item eval it would miss a difference the paired test detects. McNemar's test uses only the disagreements, which is where the information is.
The same logic tells you how to run the eval: score both versions on exactly the same items, and keep the per-item results, not just the totals. Without per-item results you can't run the paired test at all.
1. Many comparisons. Try ten prompt variants against a baseline, test each at the 5% level, and even if none is truly better there is about a 40% chance that at least one will "win". It's the same multiple-comparisons problem behind the dead salmon that read minds. Adjust for the number of comparisons (the Holm method is simple and widely available), or confirm the winner on a fresh set of items it wasn't selected on.
2. Tuning on the test. If you chose the prompt, the few-shot examples or the decoding settings by watching this eval, the winner's score is optimistic, and so is its lead. Keep a held-out set you never look at while tuning, and report the comparison on that.
3. Sampling randomness. At a temperature above zero the same item can pass on one run and fail on the next, so a single run per item mixes item difficulty with luck. Either reduce the randomness (temperature 0 or a fixed seed where the provider supports it, keeping in mind that even these aren't always fully deterministic), or run each item several times and compare the per-item average scores. The second approach needs a different test from McNemar's, such as a paired t-test on the per-item averages. Either way, say which you did.
One more check: if items come in groups, such as several questions about one document, the real uncertainty is larger than any of these tests show. Our guide to eval size explains the correction.
With enough items, even a half-point gap will be statistically significant. Whether it matters is a separate decision. Before running the comparison, write down the smallest gain that would justify switching, given cost, latency and risk, and then check the confidence interval against it. If the interval sits entirely above that line, switch. If it straddles the line, the eval can't settle the question, and more items are the fix; here's how many.
Give both accuracies, the number of items, the gap with its interval, and the test:
On the same 1,000 items, the new version was correct on 81.0% and the old version on 78.0% (difference 3.0 points, 95% CI [0.6, 5.4]; McNemar mid-p = .014).
That sentence tells a reader how big the gain is, how sure you can be of it, and that the comparison was paired. A bare "81% vs 78%" tells them none of those things.
If you're comparing models or prompts for a decision that has to hold up, such as a launch, a vendor choice or a customer's due diligence, this is the kind of evaluation work we help AI teams do.
One email whenever we publish something new. No spam, unsubscribe anytime.
{{ subscribeErrorMsg }}