LLM Benchmarks — Scores, Methodology & Live Rankings
What each benchmark measures, how models score, and the current rankings
RefCOCO-M
Vision & Language
RefCOCO-M as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): Visual grounding category, with-thinking column. RefCOCO-M variant a...
Point-Bench
Spatial Intelligence
Point-Bench as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): Visual grounding category, with-thinking column. Pointing benchmar...
MindCube (tinybench)
Spatial Intelligence
MindCube (tinybench) as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): Embodied reasoning category, with-thinking column. MindCu...
SAT (val + test)
Spatial Intelligence
SAT (val + test) as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): Embodied reasoning category, with-thinking column. SAT spatia...
RoboSpatial-Home
Spatial Intelligence
RoboSpatial-Home as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): Embodied reasoning category, with-thinking column. RoboSpatia...
Where2Place
Spatial Intelligence
Where2Place as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): Embodied reasoning category, with-thinking column. Relative spatia...
OpenEQA (episodic memory, video)
Spatial Intelligence
OpenEQA (episodic memory, video) as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): Embodied reasoning category, with-thinking co...
VSI-Bench
Spatial Intelligence
VSI-Bench as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): Embodied reasoning category, with-thinking column. Visual-Spatial In...
ScreenSpot-v2 Desktop
Vision & Language
ScreenSpot-v2 Desktop as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): GUI understanding category, with-thinking column. Screen...
ScreenSpot-v2 Mobile
Vision & Language
ScreenSpot-v2 Mobile as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): GUI understanding category, with-thinking column. ScreenS...
ScreenSpot-v2 Web
Vision & Language
ScreenSpot-v2 Web as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): GUI understanding category, with-thinking column. ScreenSpot...
ActivityNet-QA (test)
Video Understanding
ActivityNet-QA (test) as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): Video category, with-thinking column. ActivityNet-QA tes...
MotionBench (dev)
Video Understanding
MotionBench (dev) as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): Video category, with-thinking column. MotionBench dev split;...
MMMB
Multilingual
MMMB as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): Multilingual category, with-thinking column. Multilingual MMMU (MMMB); do...
Multilingual MMBench
Multilingual
Multilingual MMBench as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): Multilingual category, with-thinking column. Multilingual...
MM-ToolSandbox (partial credit)
Multimodal Agents
MM-ToolSandbox (partial credit) as evaluated on the LFM3-VL-2B card (vision benchmark, with-thinking setting): Agentic category, with-thinking column. Multim...
Decision Index (public suite)
Decision
Public decision-suite benchmark (v0.3 public suite) in multimodalart's jev-decision-index leaderboard on Hugging Face; measures binary/multi-choice decision ...
DecisionBench (Hanno Labs)
Decision
DecisionBench v1 by Hanno Labs, a decision-making benchmark suite (https://huggingface.co/datasets/Hanno-Labs/decision-bench). Values reported as percentage.
Decision Bench (decisionbench.ai)
Decision
Decision Bench v4 at decisionbench.ai, a dedicated decision-quality benchmark. Values reported as percentage. Source: https://decisionbench.ai/
Fast Decisions
Decision
Fast Decisions dataset (fastino/fast-decisions on Hugging Face), quick binary judgment tasks. Values reported as percentage. Source: https://huggingface.co/d...
JevBench
Decision
JevBench v1.4.1 (fstandhartinger/jevbench on GitHub), benchmark for judgment/decision tasks. Values reported as percentage. Source: https://github.com/fstand...
ImajevBench
Multimodal
ImajevBench v2.0-lite (mohit67890/imajev-bench on Hugging Face), image-based decision/judgment benchmark. Values reported as percentage. Source: https://hugg...
WorkflowEvals
Decision
WorkflowEvals (evals.typesafe.ai), benchmark suite over workflow/production decision steps like routing, triage, moderation. Values reported as percentage. S...
Typed Decisions
Decision
Typed Decisions dataset (LocalLLaMA/typed-decisions on Hugging Face), decision tasks with typed answers (numbers, dates, choices). Values reported as percent...
CountQA
Spatial Intelligence
CountQA, counting QA over images (counting objects in pictures as typed numeric decisions). Values reported as percentage. Source referenced on the LiquidAI/...
MMRB2
Multimodal
MMRB2, multimodal reasoning benchmark v2. Values reported as percentage. Source referenced on the LiquidAI/d1-2B model card.
MMVP
Vision & Language
MMVP, MultiModal Visual Patterns benchmark (vision patterns that trip up VLMs). Values reported as percentage. Source referenced on the LiquidAI/d1-2B model ...
NaturalBench
Multimodal
NaturalBench, benchmark for evaluating vision-language models on natural yes/no image questions. Values reported as percentage. Source referenced on the Liqu...
VisualPuzzles
Vision & Language
VisualPuzzles, visual reasoning puzzle benchmark. Values reported as percentage. Source referenced on the LiquidAI/d1-2B model card.
VisualWebBench
Vision & Language
VisualWebBench, benchmark for understanding web pages/screenshots (GUI grounding and navigation). Values reported as percentage. Source referenced on the Liq...
VL-RewardBench
Vision & Language
VL-RewardBench, benchmark for reward-style preference judgments between vision-language answers. Values reported as percentage. Source referenced on the Liqu...
BenchCAD w/o tools
Agentic Coding
BenchCAD on 1,000-case subsets randomly sampled from the 17,900-case Vision2Code dataset, max reasoning effort, single-shot generation with official prompts ...
BenchCAD w/ tools
Agentic Coding
With-tools run uses Claude Code; open models scored on 1,000-case subsets of Vision2Code.
Kimi Code Bench 3.0 Long-horizon
Agentic Coding
Internal Moonshot long-horizon coding benchmark; Kimi tested with Claude Code, competitors at max effort with possible fallback behavior.
CharXiv (RQ) w/ python
Vision & Language
CharXiv reasoning questions with python tool; averaged over 4 runs (multimodal w/ python per Kimi K3.1 Pro card footnote).
GDP.pdf w/ python
Document Understanding
Each original PDF sent to a sandbox for IPython processing; rubric graded with Gemini-3.5-Flash (medium effort); task-macro mean rubric pass rate; averaged o...
ChartoGraphy w/ python
Vision & Language
ChartoGraphy with python tool, averaged over 5 runs.
BabyVision w/ python
Vision & Language
BabyVision with python tool, averaged over 4 runs.
ZeroBench_main w/ python (pass@5)
Vision & Language
Tools-augmented ZeroBench main set, pass@5, averaged over 5 runs.
Agents' Last Exam (CLI)
General Agents
ALE v1.1 CLI subset (86 of 122 tasks); runs that bypass stated requirements score zero.
Automation Bench-AA
Agentic
Artificial Analysis-branded AutomationBench-AA eval; scores cited from artificialanalysis.ai in the Kimi K3.1 Pro card. Distinct from Automation-Bench and Au...
GDPval-AA v2.1
General Agents
Elo-scored Artificial Analysis GDPval-AA v2.1 agent benchmark; distinct from GDPVal-AA v2 (index-typed) and plain GDPVal.
OfficeQA Pro v2
Document Understanding
Newer v2 of OfficeQA Pro; PDF corpus as page images only; online answer retrieval scores zero.
Frontier Finance
Finance
Finance agent benchmark evaluated with the official Finance Agent v2 harness.
Harvey Legal Agent
General Agents
1,251 public tasks with Artificial Analysis's Stirrup harness; score is the all-pass rate over rubric criteria.
HANDBOOK.md
General Agents
Official OpenHands-based harness using tasks from the benchmark's early open-source v1 release.
OSWorld V2.1
General Agents
OSWorld v2.1 computer-use benchmark; Kimi models use the Kimi Agent harness in the official OSWorld v2 repo.
Business Operation
Agentic
Internal Moonshot agentic benchmark for business operation tasks.