LLM Benchmarks — Scores, Methodology & Live Rankings
What each benchmark measures, how models score, and the current rankings
AgentWorldBench Android
General Agents
AgentWorldBench Android domain - open-ended rubric evaluation for language world models simulating Android device environments. Normalized to 0-100.
AgentWorldBench MCP
General Agents
AgentWorldBench MCP domain - open-ended rubric evaluation (5 dimensions: Format, Factuality, Consistency, Realism, Quality) for language world models simulat...
AgentWorldBench OS
General Agents
AgentWorldBench OS domain - open-ended rubric evaluation for language world models simulating operating-system environments. Normalized to 0-100.
AgentWorldBench Search
General Agents
AgentWorldBench Search domain - open-ended rubric evaluation for language world models simulating search environments. Normalized to 0-100.
AgentWorldBench SWE
General Agents
AgentWorldBench SWE domain - open-ended rubric evaluation for language world models simulating software engineering environments. Normalized to 0-100.
AgentWorldBench Terminal
General Agents
AgentWorldBench Terminal domain - open-ended rubric evaluation for language world models simulating terminal environments. Normalized to 0-100.
AgentWorldBench Web
General Agents
AgentWorldBench Web domain - open-ended rubric evaluation for language world models simulating web browser environments. Normalized to 0-100.
AndroidWorld
General Agents
AndroidWorld - visual agent benchmark for Android device task completion.
DynaMath
Vision & Language
DynaMath - dynamic mathematical reasoning in visual contexts.
MVBench
Video Understanding
MVBench - comprehensive multi-modal video understanding benchmark.
OCRBench
Document Understanding
OCRBench - comprehensive OCR capability evaluation benchmark.
VlmsAreBlind
Vision & Language
VlmsAreBlind - benchmark testing visual perception capabilities that reveal blind spots in VLMs.
SWE Atlas - RF
Agentic Coding
SWE Atlas RF - agentic coding subtask; mini SWE agent harness, temp=1.0, top_p=0.95, 128K context, avg of 5 runs.
SWE Atlas - TW
Agentic Coding
SWE Atlas TW - agentic coding subtask; mini SWE agent harness, temp=1.0, top_p=0.95, 128K context, avg of 5 runs.
TauBench V3 Airline
General Agents
TauBench V3 - Airline domain. Agentic tool use benchmark for airline customer service.
TauBench V3 Retail
General Agents
TauBench V3 - Retail domain. Agentic tool use benchmark for retail customer service.
TauBench V3 Telecom
General Agents
TauBench V3 - Telecom domain. Agentic tool use benchmark for telecom customer service.
TauBench V3 Banking
General Agents
TauBench V3 - Banking domain. Agentic tool use benchmark for banking customer service.
TauBench V3 Average
General Agents
TauBench V3 - Average across all domains (Airline, Retail, Telecom, Banking).
IOI 2025
STEM & Reasoning
IOI 2025 - International Olympiad in Informatics 2025 competitive programming score.
IMOAnswerBench (with tools)
STEM & Reasoning
IMOAnswerBench - with tools variant. International Mathematical Olympiad answer evaluation benchmark with tool use.
CritPt (no tools)
STEM & Reasoning
CritPt - critical point identification benchmark, no tools variant.
Longbench v2 (≤ 1M)
Long Context
Longbench v2 (≤ 1M) - long context understanding benchmark, up to 1M tokens.
DeepSWE 1.1
Agentic Coding
DeepSWE 1.1 - agentic coding benchmark evaluated with Claude Code harness at temp=1.0, top_p=0.95, 256K context window.
QwenSWEBench
Agentic Coding
QwenSWEBench - in-house coding benchmark for evaluating software engineering capabilities. Evaluated with Claude Code harness, avg@3, 8-hour timeout, max_tok...
CoWorkBench
General Agents
CoWorkBench - in-house cowork benchmark for evaluating long-horizon tasks across computer science, finance, law, medical, and other productivity domains.
HLE (with tools)
STEM & Reasoning
HLE (with tools) - Humanity's Last Exam with tool use variant.
MMLU-ProX
Multilingual
MMLU-ProX - multilingual extended MMLU benchmark, average across en/de/fr/es/it/ja/zh/hi/pt/ko.
Multi-Challenge
Instruction Following
Multi-Challenge - multi-turn instruction following benchmark by ScaleAI.
SciCode (subtask)
STEM & Reasoning
SciCode (subtask) - scientific code generation benchmark, subtask-level scoring.
WMT24++ (en→xx)
Multilingual
WMT24++ (en→xx) - machine translation benchmark, English to other languages.
Apex-Shortlist (no tools)
STEM & Reasoning
Apex-Shortlist - no tools variant. Competitive math benchmark without tool use.
OmniScience Accuracy
Knowledge
OmniScience Accuracy - scientific knowledge accuracy benchmark.
Apex-Shortlist (with tools)
STEM & Reasoning
Apex-Shortlist - with tools variant. Competitive math benchmark with tool use.
Vals.ai Financial Agent 1.1 (without web search)
General Agents
Vals.ai Financial Agent 1.1 - without web search mode. Financial agentic benchmark.
Vals.ai Financial Agent 1.1 (with web search)
General Agents
Vals.ai Financial Agent 1.1 - with web search mode. Financial agentic benchmark.
BrowseComp
General Agents
BrowseComp - web browsing and comprehension agentic benchmark.
WebArena-Verified
General Agents
WebArena-Verified - browser use benchmark evaluating autonomous web navigation and task completion. Scores computed with official WebArena-Verified grader un...
RecreationBench
General Agents
RecreationBench - in-house long-horizon application-recreation benchmark evaluating hybrid-agent capabilities across five platforms: desktop (Ubuntu, macOS, ...
SWE-MM
Agentic Coding
SWE-MM - multimodal software engineering benchmark. Evaluated on Claude Code harness using public dev split of SWE-bench Multimodal with modifications.
Vision2Web
Vision & Language
Vision2Web - visual web development benchmark. Scores averaged across frontend, webpage, and website categories. Evaluated with Claude Code harness, judged b...
NL2Repo-Bench
Agentic Coding
NL2Repo-Bench - repository generation from natural language. Evaluated with Claude Code harness. Bash commands that access the specific repository are disabl...
FrontierSWE
Agentic Coding
FrontierSWE - agentic coding benchmark. MEAN@5 scores from the official FrontierSWE leaderboard. Dominance scores recomputed from raw scores using the offici...
MLS-Bench-Lite
Agentic Coding
MLS-Bench-Lite - agentic coding benchmark. Evaluated with Claude Code using a 5-hour timeout and max_tokens=131,072. Scores from the official leaderboard.