Arena-Hard v2

Arena-Hard v2 measures chat and instruction quality as a win rate: two models answer the same prompt and a judge model decides which answer is better, mirroring LMArena-style preference ratings. The score says something about perceived answer quality — not about provable correctness.

Arena-Hard v2 measures how precisely a model obeys the instructions it was given. In detail, it checks:

• helpful, well-structured chat answers
• relative quality compared against a fixed reference model
• consistency of judgement by a fixed judge model
• real-world preference rather than exam-style correctness

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the Arena-Hard v2 benchmark?

Arena-Hard v2 measures chat and instruction quality as a win rate: two models answer the same prompt and a judge model decides which answer is better, mirroring LMArena-style preference ratings. The score says something about perceived answer quality — not about provable correctness.

What does Arena-Hard v2 test?

Arena-Hard v2 mainly measures how precisely a model obeys the instructions it was given. Concretely, it checks: helpful, well-structured chat answers; relative quality compared against a fixed reference model; consistency of judgement by a fixed judge model.

How is the Arena-Hard v2 score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get Arena-Hard v2 data and scores?

The official source is https://huggingface.co/lmarena-ai/arena-hard-v2. The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/arena-hard-v2/scores) or pull it directly from the Open LLM Radar API with any HTTP client.