Is model A really better than model B?

When two models answer the same eval items, a gap in accuracy can be real or just luck of the draw. Enter how often each model got items right and you get the gap with a confidence interval and a paired significance test.

Your eval results

Model A name

Model B name

How do you want to enter results?

Number of items in each cell. Every item is scored right or wrong for both models.

Model B rightModel B wrong Model A right Model A wrong

The shaded cells are the items where the models disagree. They decide the test.

Confidence level

Result

Summary you can paste into a report

Why a two-point lead often proves nothing

Suppose model A scores 87.0% on a 200-item eval and model B scores 84.5%. It looks like A is better. But the two models agreed on 177 of the 200 items, both right or both wrong, and those items say nothing about which is better. The comparison rests on the 23 items where they disagreed: A alone was right on 14, B alone on 9. A 14-to-9 split is well within what you would see if the models were equally good (mid-p = .31), and the 95% interval for A's true advantage runs from about −2.3 to +7.2 points. This eval cannot tell the two models apart. On a 500-item eval with a similar gap, 83.0% against 79.4% and a 38-to-20 split of the disagreements, it can (mid-p = .018). The calculator above starts with the 200-item example; replace it with your own counts.

Why the test has to be paired

Both models answered the same items, so their scores are not independent: easy items tend to be easy for both. A test that treats the two accuracies as separate samples, such as a two-proportion z-test, ignores that and usually overstates the uncertainty, hiding real differences. The paired approach, McNemar's test, looks only at the items where the models disagree, which is where the information is.

What the calculator reports

The accuracy gap, A minus B, in percentage points.
A confidence interval for the gap, using the Agresti–Min method: add half an item to each cell, then build an ordinary interval. It behaves well even with few disagreements.
A McNemar mid-p test, which asks how surprising the split of disagreements would be if the models were equally good. Fagerland and colleagues (2013) found the mid-p version keeps close to its nominal error rate in small and large samples, while the classic exact version is overly cautious. The exact and chi-square versions are listed for reference.

Before you trust the result

Items must be independent. If items come in groups, such as several questions about one document or paraphrases of one prompt, the real uncertainty is larger than shown. Clustered standard errors are the fix.
Randomness in the models counts too. If outputs are sampled at a temperature above zero, one run per item mixes item difficulty with luck. Run each item several times, or fix the sampling, and say which you did.
Many comparisons inflate false wins. Compare ten model variants against a baseline and one will probably "win" by chance. Adjust for the number of comparisons, or confirm the winner on fresh items. This is the multiple comparisons problem.
A tuned-on eval flatters. If you picked prompts or settings by looking at this eval, its score is optimistic. Hold out items you never looked at.
Significant is not the same as important. With thousands of items a 0.5-point gap can be "significant". Decide in advance what gap would change your decision, and check the interval against it.

Checked against R and Python

The calculations run in your browser and were compared case by case with R (binom.test, mcnemar.test, qnorm) and with SciPy (binomtest, chi2.sf, norm.ppf) across 369 scenarios, from 10 to 20,000 items. Every p-value agreed to within 1 part in a billion and every interval bound to within 10⁻¹². The code panel above reproduces your result in either language.

Sources

Agresti, A., & Min, Y. (2005). Simple improved confidence intervals for comparing matched proportions. Statistics in Medicine, 24(5), 729–740. doi:10.1002/sim.1781
Fagerland, M. W., Lydersen, S., & Laake, P. (2013). The McNemar test for binary matched-pairs data: mid-p and asymptotic are better than exact conditional. BMC Medical Research Methodology, 13, 91. doi:10.1186/1471-2288-13-91
Miller, E. (2024). Adding error bars to evals: A statistical approach to language model evaluations. arXiv:2411.00640

Common questions

Why can't I just compare the two accuracy scores?

Both models answered the same items, so their scores are not independent. Only the items where they disagree carry information. In the page's example the models agreed on 177 of 200 items, and a 14-to-9 split on the rest (mid-p = .31) is well within what equally good models would produce.

Which test does the calculator use?

McNemar's test, reported as a mid-p value (the exact and chi-square versions are listed for reference), plus an Agresti–Min confidence interval for the accuracy gap. Fagerland and colleagues (2013) found the mid-p version stays close to its nominal error rate in small and large samples.

How many items do I need?

It depends on the gap and on how often the models disagree. In the page's example, a 200-item eval could not separate scores of 87.0% and 84.5%, while a 500-item eval with a similar gap and a 38-to-20 split could (mid-p = .018). See How many test items does an AI eval need?

What can make the result unreliable?

Items that are not independent (such as several questions about one document), sampled outputs with only one run per item, many comparisons that inflate false wins, and prompts or settings tuned on the same eval. Hold out items you never looked at.

Is my data sent anywhere?

No. The calculations run in your browser, and nothing you enter or paste is sent anywhere or stored.

How was the calculator checked?

Its calculations were compared case by case with R and SciPy across 369 scenarios, from 10 to 20,000 items. Every p-value agreed to within 1 part in a billion.