Introducing AutoEval to the Arena leaderboards

AutoEval uses the reward model in place of human votes, then follows the exact same Arena ranking methodology to compute the final arena score. Specifically, for every target model it runs four steps:

1. Live Eval Sampling — sample real prompt-and-response pairs $(p, r_A, r_B)$ from live evaluation.
2. Model Generation — regenerate response B from the target model on the same prompt.
3. RM Pairwise Vote — the reward model scores each response, applying a softmax to the score differences to obtain a soft probability vote.
4. AutoEval Score — combine the autorater’s votes for the target model with the live human votes for other models to produce the Arena Score.

So, how close is AutoEval to the real thing? To test this, we trained a reward model using data up to the end of April 2026 and evaluated AutoEval scores for all new models released after that date. When comparing these to live evaluation scores, the results (shown in the figure below) demonstrate that AutoEval scores are highly correlated with live evals. Specifically, the ranking correlation exceeds 0.98, and most AutoEval scores fall well within the confidence intervals of the live scores.

We also tested AutoEval’s ability to correctly choose the better model in head-to-head comparisons across >40 test models. AutoEval achieves >90% accuracy when the true performance gap exceeds 10 points, and 100% accuracy beyond 15 points. When models are within 5 points of each other, they are statistically indistinguishable under live evaluation due to overlapping confidence intervals.

This high correlation demonstrates that AutoEval is accurate enough to drive critical model selection decisions. For example, it can be used to shortlist the most promising models from a large pool before committing resources to a full live evaluation.

AutoEval isn’t text-only. We apply the same methodology across modalities — text, vision, image generation, and code arenas.

Taking image generation as an example, we trained a text-to-image reward model on over 3 million preference pairs. By training on this massive preference dataset—built from realistic image generation prompts requested by real-world users—our reward model not only excels on our own distribution but also achieves top-tier performance on public benchmarks. Evaluated on MMRB2, Meta’s 2026 multimodal reward benchmark, Arena-RM achieves state-of-the-art compared with other pointwise text-to-image reward models, more than 9 points ahead of the runner-up (HPSV3, UnifiedReward, PickScore, VQAScore and others).

Arena leaderboards remain a public good and AutoEval makes that value available sooner: right at a model's release. It already works across Text, Vision, Image, and Code Arena, providing the ability to rank new models in hours rather than days, while focusing on the domains the community cares about. As we continue working through open challenges in reward modeling, we’ll be extending this approach to even more modalities and incorporating more signals to better capture human preferences.