Is model A really better than model B?

When two models answer the same eval items, a gap in accuracy can be real or just luck of the draw. Enter how often each model got items right and you get the gap with a confidence interval and a paired significance test.

Your eval results

Model A name

Model B name

How do you want to enter results?

Number of items in each cell. Every item is scored right or wrong for both models.

Model B rightModel B wrong Model A right Model A wrong

The shaded cells are the items where the models disagree. They decide the test.

Confidence level

Result

Summary you can paste into a report

Why a two-point lead often proves nothing

Suppose model A scores 87.0% on a 200-item eval and model B scores 84.5%. It looks like A is better. But the two models agreed on 177 of the 200 items, both right or both wrong, and those items say nothing about which is better. The comparison rests on the 23 items where they disagreed: A alone was right on 14, B alone on 9. A 14-to-9 split is well within what you would see if the models were equally good (mid-p = .31), and the 95% interval for A's true advantage runs from about −2.3 to +7.2 points. This eval cannot tell the two models apart. On a 500-item eval with a similar gap, 83.0% against 79.4% and a 38-to-20 split of the disagreements, it can (mid-p = .018). The calculator above starts with the 200-item example; replace it with your own counts.

Why the test has to be paired

Both models answered the same items, so their scores are not independent: easy items tend to be easy for both. A test that treats the two accuracies as separate samples, such as a two-proportion z-test, ignores that and usually overstates the uncertainty, hiding real differences. The paired approach, McNemar's test, looks only at the items where the models disagree, which is where the information is.

What the calculator reports

The accuracy gap, A minus B, in percentage points.
A confidence interval for the gap, using the Agresti–Min method: add half an item to each cell, then build an ordinary interval. It behaves well even with few disagreements.
A McNemar mid-p test, which asks how surprising the split of disagreements would be if the models were equally good. Fagerland and colleagues (2013) found the mid-p version keeps close to its nominal error rate in small and large samples, while the classic exact version is overly cautious. The exact and chi-square versions are listed for reference.

Before you trust the result

Items must be independent. If items come in groups, such as several questions about one document or paraphrases of one prompt, the real uncertainty is larger than shown. Clustered standard errors are the fix.
Randomness in the models counts too. If outputs are sampled at a temperature above zero, one run per item mixes item difficulty with luck. Run each item several times, or fix the sampling, and say which you did.
Many comparisons inflate false wins. Compare ten model variants against a baseline and one will probably "win" by chance. Adjust for the number of comparisons, or confirm the winner on fresh items. This is the multiple comparisons problem.
A tuned-on eval flatters. If you picked prompts or settings by looking at this eval, its score is optimistic. Hold out items you never looked at.
Significant is not the same as important. With thousands of items a 0.5-point gap can be "significant". Decide in advance what gap would change your decision, and check the interval against it.

Checked against R and Python

The calculations run in your browser and were compared case by case with R (binom.test, mcnemar.test, qnorm) and with SciPy (binomtest, chi2.sf, norm.ppf) across 369 scenarios, from 10 to 20,000 items. Every p-value agreed to within 1 part in a billion and every interval bound to within 10⁻¹². The code panel above reproduces your result in either language.

Sources

Agresti, A., & Min, Y. (2005). Simple improved confidence intervals for comparing matched proportions. Statistics in Medicine, 24(5), 729–740. doi:10.1002/sim.1781
Fagerland, M. W., Lydersen, S., & Laake, P. (2013). The McNemar test for binary matched-pairs data: mid-p and asymptotic are better than exact conditional. BMC Medical Research Methodology, 13, 91. doi:10.1186/1471-2288-13-91
Miller, E. (2024). Adding error bars to evals: A statistical approach to language model evaluations. arXiv:2411.00640

Building an eval you need to defend to a customer, an auditor or your own leadership? DASS helps AI teams design evaluations with the right sample sizes, uncertainty and comparisons.