Arena Elo and Preference Ranking

Static benchmarks answer questions somebody wrote down in advance. Chatbot Arena answers a different one: given a real prompt from a real user, which of two anonymous models did they prefer? Votes are collected pairwise, models are unlabelled during the vote, and the aggregate becomes a ranking (Chiang et al., ICML 2024, arXiv:2403.04132).

This solves the contamination problem, because the prompts are fresh, and creates a set of new ones.

Bradley-Terry, not Elo

The leaderboard displays Elo-style numbers, but the estimator is Bradley-Terry. The distinction matters. Under BT, each model \(i\) has a latent strength \(\beta_i\) and

The \(\beta\) are fit by maximum likelihood over the entire vote history at once. Classical Elo, by contrast, is an online update rule whose result depends on the order games were played in, which is a bug when you are trying to rank a fixed set of models from a fixed pool of votes. Fitting BT globally makes the ranking order-invariant and lets you put a proper confidence interval on each \(\beta\) via bootstrap.

The scores are also only identified up to an additive constant, which is why an arena score is meaningful only as a difference. A 15-point gap between adjacent models corresponds to roughly a 52 percent win rate, which is close to a coin flip.

What the arena measures

Human preference over a prompt distribution that is not yours. Arena traffic skews toward short creative and conversational prompts, and votes are cast quickly by users who mostly do not verify factual claims. The reliable consequence is that presentation is rewarded: formatting, length, confident tone, and markdown structure all correlate with winning. Length in particular is strong enough that the Arena team publishes style-controlled variants of the leaderboard that regress out response length and markdown usage.

That is not a flaw so much as a scope statement. Arena rank is a good predictor of which model people enjoy talking to and a poor predictor of which model gets the right answer on your domain.

Selection effects

The most serious critique is structural rather than statistical. Providers can test many private variants before releasing a public score and retract results they dislike; taking the maximum over \(k\) private draws inflates the published score even when every draw is unbiased. An audit identified 27 private variants tested by a single provider in the run-up to one model release, alongside sampling-rate and deprecation asymmetries that favour proprietary models (Singh et al., 2025, The Leaderboard Illusion, arXiv:2504.20879).

The lesson generalises to any internal leaderboard: if a team can run the eval repeatedly and choose when to publish, the published number is a maximum, not a mean.

When it breaks

Vote sparsity in the tail. New models and rarely-served models have wide intervals that the ranked table hides. Two models 8 points apart with 95 percent intervals of ±12 are not ranked; they are adjacent.

Non-transitivity. BT assumes a single latent strength dimension. Real preferences cycle: A beats B on coding, B beats C on prose, C beats A overall. Forcing a total order on genuinely multi-dimensional preferences discards exactly the information you needed.

Optimising for the arena is easy and hollow. Train on longer, more formatted, more agreeable outputs and rank improves without capability improving. This is sycophancy with a leaderboard attached.