Factuality in the Arena

We can plot each model’s score against the factuality weight. Weight 100% means that the score is purely factuality-based, 0% means it is purely human-preference-based. Different models have different slope. Most top models (by human preference score) have decreasing score as factuality weight is increased. On both Text and Search Arena we see similar trends: OpenAI's GPT and SpaceXAI's Grok models go up or stay flat as factuality weight increases. Anthropic's Claude and Google's Gemini models go down or stay flat. Finally, Meta's muse-spark-1.1 and Baidu's ernie-5.1 drop heavily.

Above we show factuality weight vs. score for all models broken down by four providers to visualize patterns in each provider's model suite. We see most OpenAI models move up in score while Anthropic models decrease or stay the same. In particular, the previous most factual Claude Opus version before the newest 4.8 release was 4.5—released back in November 2025. We find that while Anthropic models are generally more preferred by humans, OpenAI's newest models are generally more factual.

SpaceXAI's Grok-4-0709 was a solidly factual model; however, the subsequent Grok-4.1 series heavily regressed in factuality. The newer Grok-4.20, Grok-4.3, and Grok-4.5 versions have improved their factuality, though Grok-4.3 is far less preferred by humans.

Google models have less variance in factuality scores OpenAI and SpaceXAI models. Notably, the Gemini-2.5 model series still remain the most factual to this day—Gemini seems to be getting less factual over time.

These trends remain similar when models have access to search tools in the Search Arena.

We find most open source models decline in score under higher factuality weight, with the exceptions of mistral-medium-3.5 and Tencent's huyuan-hy3-preview.

While human preference across nearly all providers has been climbing over time, plotting pure factuality score against flagship model release date shows OpenAI is the only provider steadily improving factuality at the same time. SpacexAI has also been improving with their recent releases; this is similarly true in Search Arena. Notably, Google and Anthropic models have stayed at relatively similar levels of factuality across the past year, especially when given access to search tools. Meta showed a large improvement in factuality between muse-spark and muse-spark-1.1 but still remains behind other top closed and open source labs.

We can also observe the relationship between human preference score and factuality score. We notice a weak positive correlation. Ultimately, the signals are largely orthogonal—this is why adding a factuality signal is a necessary subcomponent of evaluation. For example, a model response is perfectly factual if the model refuses to provide an informative response; such responses perform poorly in terms of human preference. At the same time, a long response with many facts may seem comprehensive and win a human preference vote, but it may also introduce factual errors.

Looking at average claim correctness for various industry categories, we find slight, but intuitive, variance in model factuality in different industries. Models are largely factual in mathematical tasks. They are also reasonably factual in Software, Medical, and Scientific tasks. The least factual industry area was Legal and Government tasks.

Below we provide a viewer to show a sample of extracted claims.

Rewarding style is not an incorrect thing to do. Communicating effectively with humans is essential in chat, so formatting and prose are important factors in evaluation. The deeper worry is that a more “stylish” response will obfuscate undeniable errors in a model's response.

With the new factuality leaderboard, we address this concern head-on: rather than using style as a proxy for such undesirable behavior, we attempt to measure and detect the existence of such behavior directly. Factuality is, of course, only one notion of response correctness, but it is a particularly important and appropriate one to prioritize because: (1) humans are increasingly trusting models to answer questions outside their area of knowledge; (2) increasingly models themselves are better at checking facts than humans.

Below we provide a more detailed explanation of the methods used to fairly audit claims and ingest them into a principled leaderboard calculation.

Composite Bradley-Terry model

Rather than relying on either human preference or factuality alone, we use the composite Bradley-Terry model to combine both objectives. While factuality measures whether a response is factually sound, it alone does not capture whether it addresses what the user actually asked for. The human preference component preserves that signal, promoting responses that are both factually grounded and responsive to user intent.

Composite loss

We fit one rating vector $\theta \in \mathbb{R}^M (M$ = number of models on the leaderboard) by minimizing a composite loss with two components plus a small ridge penalty:

• $L_\text{human}(\theta)$ is a standard Bradley-Terry negative log-likelihood evaluated on per-battle human-vote outcomes.
• $L_\text{fact}(\theta)$ is a standard BT negative log-likelihood evaluated on per-battle factuality-derived soft outcomes.
• Note: $w_\text{human} + w_\text{fact} = 1$. We will most often write the factuality weight as just $w; w_\text{human} = 1 - w$.

The fitted rating $\theta$ is the rating that best explains the joint distribution of human votes and factuality outcomes simultaneously, weighted by $w$.

Factuality outcome per battle

For battles eligible for the factuality term, the factuality outcome is a soft outcome in $(0,1)$ produced by applying a sigmoid to the per-side aggregate truth-probability gap:

• $\text{avg\_truth\_prob}_x$ is the mean of the per-claim aggregated truth probabilities across all verified claims emitted by side $x$ of the battle.
• $T$ is a fixed temperature chosen so that the variance of the factuality-vote distribution matches the variance of the human-vote distribution.

Which battles enter the factuality term

A battle is eligible to contribute to $L_\text{fact}$ whenever at least one side emits a web verifiable claim. We find that roughly ~76% of Text Arena battles have relevant verifiable claims. The handling per case:

Abstention is never penalized, but emitting false claims is — even when the opposing side abstained. We expect $L_\text{human}$ to counterweight this signal correctly; the composite system inherently disciplines pure abstention via $L_\text{human}$. The factuality axis is purely meant to capture the harm of confident misstatements.

Notice we only care about verifiable claims. These are claims that are web verifiable, unambiguous, and objective. Claims that do not fall under this are not extracted and are skipped.

Examples:

Confidence intervals

Per-model rating confidence intervals are computed via the sandwich variance estimator, as they are on the previous leaderboards. In particular, letting $\hat{\theta} = \operatorname*{argmin}_{\theta} L(\theta)$, we have:

where $H = \sum_k w_k \cdot H_k$​ is the composite Hessian ($H_k$​ denotes either the human preference or factuality BT Hessian), $B = \sum_k \left( w_k^2 / N_k \right) \cdot \Sigma_k$ is the composite gradient covariance, and $N_k$​ denotes the corresponding total number of battles. Reported intervals on the leaderboard are 95% intervals derived from this estimator, namely $(\hat{\theta}_m \pm 1.96\sqrt{\widehat{\text{Var}}(\hat{\theta})_{mm}})$ for model $m$, mapped into the standard Elo-like rating space.

Confidence interval widths shift as $w$ changes—in particular, intervals tend to widen at higher ww when factuality data is sparser than human-vote data.