When a new model tops a public leaderboard, the headline writes itself: first place. The ranking that produces that headline comes from one of the most widely watched AI evaluations, Chatbot Arena, run by the LMSYS research group and now published as LMArena. People type a prompt, two anonymous models answer, and the person votes for the answer they prefer. Over a million of those pairwise votes become a score for each model, and the scores become a ranked list.
The people who run the leaderboard are unusually open about what that ranking can and cannot say. Their method fits a Bradley-Terry model to the votes, which estimates how likely each model is to beat each other model, and they attach a confidence interval to every score, estimated by resampling the votes. Ties between the two answers count as half a win for each model. Then comes the part that gets lost when the list is quoted in a headline: the leaderboard reports not just a rank but a range of ranks. A model's best possible rank is one plus the number of models whose entire confidence interval sits above its own. Two models whose intervals overlap cannot be separated with the votes collected so far, so the data do not show that one beats the other.
Here is an illustration with made-up numbers, not a real snapshot. Suppose five models score 1,480, 1,477, 1,474, 1,471 and 1,455, each with an interval of about plus or minus 6 points. The first four overlap with each other. The ranking calls them first, second, third and fourth, but the honest reading is "one group of four, with the fifth model clearly behind." Anything you could say about who is first is noise from which votes happened to arrive. The more models bunch together at the top, which is what happens in a crowded field, the more the top of the list turns into a tie rather than a podium.
The LMArena team has changed how it computes these intervals over time, and the changes affect how many ties appear. A published methodology note describes moving to a closed-form approximation for the intervals and reweighting votes, and says the result is more ties from overlapping intervals, especially where fewer votes are available per model, as in the vision arena. A change like that does not make the models any worse or better. It makes the leaderboard more honest about what the votes can support.
Do not ask who is first. Ask which models are separated from which, and by how much compared with the uncertainty. When a leaderboard gives you a range of ranks, use it. When it gives only a single number, ask how many comparisons stand behind it and whether the gap to the next model is larger than the interval around it. If two models you are choosing between are inside each other's intervals, decide on something the leaderboard does not measure: your own task, your cost, your data.
If a ranking is about to drive a decision and you cannot tell whether the gaps are real, send us the table. Working out which differences hold up is usually a short conversation.
One email whenever we publish something new. No spam, unsubscribe anytime.
{{ subscribeErrorMsg }}