Part of our AI Eval Watch series · Oct 28–Nov 3, 2026
This week's AI-evaluation preprints kept returning to one uncomfortable question: when the grader changes, does the answer change too? Below are four preprints, three about LLM judges and one about the rule-based scorers inside standard benchmarks. All four are not peer reviewed, and each item says what its authors report in their abstract, not something we have independently verified.
The authors asked whether a second LLM judge from a different provider can stand in for human labels when you want to check a conclusion. In a preregistered study, five judge models from three providers re-scored 4,000 Chatbot Arena battles, each judged twice with the two responses in both orders. Verdict-level agreement with the human vote was low, with a median Cohen's kappa of 0.22. How closely each judge's system ranking tracked the human ranking varied widely, with Kendall's tau from 0.17 to 0.89 (median 0.67), so the preregistered prediction of stable rankings was not supported. Requiring two inexpensive judges from different providers to agree raised precision by 0.10, but the authors attribute most of that gain to dropping one badly aligned judge, and the paired rule produced a conclusion for only 35% of comparisons. Judges that did not track humans failed to settle on a winner for 26% to 70% of system pairs at up to 100 verdicts per pair. Judge errors also clustered together far more than chance would predict, and with only one human vote per battle the authors say they cannot tell shared judge bias apart from noise in the human votes.
Why it matters. This is a direct test of validity, with the human vote as the reference, and the result is sobering: a kappa of 0.22 is weak agreement, and a ranking that can range from weak to strong agreement with humans depending on which judge you pick is not a property of the systems being ranked. The preregistration is a strength, because the null result was stated in advance rather than discovered after the fact. The correlated errors are the key caveat. If judges fail together, a second judge that agrees with the first adds less independent evidence than it seems to, and agreement between judges is consistency, which is necessary but not sufficient. The human reference has its limits too, since one crowd vote per battle is itself a noisy label. Our guide on telling whether one LLM is really better than another covers why a ranking needs an uncertainty estimate before anyone acts on it.
Multiple-choice benchmarks for vision-language models still need a rule that turns a model's free-text reply into a right or wrong mark, and the authors argue that this rule rarely gets audited. They re-ran the official scorers of eight peer-reviewed benchmarks on five open models small enough for a single 16 GB GPU, and built a blinded adjudication that combines human and LLM judges to recover what each model actually answered, with agreement of kappa 0.956. Holding the model outputs fixed, swapping the scoring rule changed accuracy by as much as 22 points. When a model replied with a clean option letter, the rule made no difference. Even a reasoning model with no off-format replies still moved 9.5 points depending on which regular-expression match the parser kept. The authors recommend that benchmarks report how often replies fall outside the expected format, and report results under at least two scoring rules.
Why it matters. A 22-point swing on identical outputs is measurement error that comes entirely from the instrument, not from the model, and it can easily exceed the gap between two models on a leaderboard. The adjudication step is the validity check here: a carefully built human-plus-judge reference, with very high agreement, against which the official parsers can be compared. Its limit, as with any reference standard, is that it is only as good as the people and rules behind it. For anyone running these benchmarks in-house, the practical point is to treat the scorer as part of the eval design, document it, and check a sample of its verdicts by hand. Our guide on what makes an AI eval defensible to auditors covers documenting scoring rules and checking them against a human baseline.
Pooling several LLM judges is a common way to steady their scores, but the authors point out that judges can agree because they share the same pull toward length, formatting or fluency, not because they are tracking the quality you care about. Their method, VoirDire, makes controlled edits to those surface features, measures how much each judge's score moves in response, and gives less weight to the judges that move most. They also give theoretical results showing that judge outputs on their own cannot separate agreement about the target quality from agreement driven by nuisance features. Across 12 datasets spanning continuous scores, binary labels and pairwise preferences, they report up to 26.8% lower mean absolute error on continuous-score tasks and up to 5.3 percentage points higher accuracy on the other two task types, compared with existing aggregation baselines.
Why it matters. The theoretical result is the most useful part for practitioners: it says, in formal terms, that you cannot certify a panel of judges as valid just by showing that they agree. That is the reliability-versus-validity distinction applied to aggregation. The intervention approach adds a different kind of evidence, sensitivity to features that should not matter, which is closer to a validity check than inter-judge agreement is. The reported gains are maxima across tasks, though, and the abstract does not say how much they vary from one dataset to the next. Our guide on evaluating an LLM beyond average accuracy covers why a single summary number can hide this kind of variation.
Judge scores on a 1-to-5 or 1-to-10 scale are ordinal ratings, and the authors argue that calibration should treat them as a distribution rather than as isolated numbers. They benchmark streaming quantile matching, a classical technique they say had not been tested for this task, which maps the judge's score distribution onto the distribution of human scores in a small human-labeled anchor set. According to the abstract, unlike point-by-point calibration methods, it preserves the ordering of the human anchor ratings and ranked best overall on distributional fidelity across the models and data tested. They also report that calibrating after the fact shrinks the raw error gap between small and large judge models. The abstract gives no numbers and does not say how many models or datasets were used.
Why it matters. Calibration against a human anchor set is a modest form of validity evidence, because the judge's scores are being tied to human ratings rather than to other judges. But matching the shape of a distribution is not the same as getting individual items right: a calibrated judge can reproduce the human score distribution while still ranking particular responses differently from people. The anchor set also has to be large and representative enough to estimate the human distribution well. Our guide on how many test items an eval needs gives a sense of how quickly precision improves with sample size.
If you are building evaluation metrics for an AI product and need them to hold up under real scrutiny, this is squarely what we do. For a quick paired comparison of two models scored on the same items, our free model-comparison calculator runs the test.
One email whenever we publish something new. No spam, unsubscribe anytime.
{{ subscribeErrorMsg }}