LLM Benchmarks — Scores, Methodology & Live Rankings
What each benchmark measures, how models score, and the current rankings
FinEntity
Finance
FinEntity: financial entity extraction/classification in finance text, scored as macro-F1 on a 0-100 scale in Cloudflare's internal Decision Index 0.2.1 suite.
VAST
Vision & Language
VAST: visual-audio scene/text classification variant, scored as macro-F1 on a 0-100 scale in Cloudflare's internal Decision Index 0.2.1 suite.
NLI4CT
Document Understanding
NLI4CT: natural-language inference over clinical trial reports, scored as macro-F1 on a 0-100 scale in Cloudflare's internal Decision Index 0.2.1 suite.
CRUXEval
Code
CRUXEval: code reasoning (output prediction and input execution over short Python snippets), scored as accuracy on a 0-100 scale in Cloudflare's internal Dec...
CLadder
STEM & Reasoning
CLadder: causal reasoning benchmark based on structural causal models, scored as accuracy on a 0-100 scale in Cloudflare's internal Decision Index 0.2.1 suite.
ForecastBench
Reasoning
ForecastBench: LLM forecasting benchmark scored with a Brier score where LOWER is better (0 = perfect calibration, larger = worse). Scores here are Brier sco...
Habermas Machine
Reasoning
Habermas Machine: AI-mediated group deliberation/consensus benchmark, scored as accuracy on a 0-100 scale in Cloudflare's internal Decision Index 0.2.1 suite.
PhishNChips
Cybersecurity
PhishNChips: phishing-detection benchmark, scored as accuracy on a 0-100 scale in Cloudflare's internal Decision Index 0.2.1 suite.
RAGTruth
Safety
RAGTruth: hallucination detection in retrieval-augmented generation outputs, scored as hallucination F1 on a 0-100 scale in Cloudflare's internal Decision In...
HoVer
Knowledge
HoVer: multi-hop fact verification over claims requiring reasoning over Wikipedia documents, scored as accuracy on a 0-100 scale in Cloudflare's internal Dec...
When2Call MCQ
Agentic
When2Call MCQ: multiple-choice evaluation of when to call a tool vs answer directly, scored as accuracy on a 0-100 scale in Cloudflare's internal Decision In...
New Yorker
Composite
New Yorker: caption/content comprehension eval tied to New Yorker cartoon caption material, scored as accuracy on a 0-100 scale in Cloudflare's internal Deci...
HumanEval+
Code
HumanEval+ is the EvalPlus extension of HumanEval with 80x more generated unit tests for correctness checking. Score recorded from the Bonsai 2 27B GGUF card...
MBPP+
Code
MBPP+ is the EvalPlus extension of Mostly Basic Python Problems with hardened test suites. Score recorded from the Bonsai 2 27B GGUF card's Full Per-Benchmar...
OCR Bench v2
Document Understanding
OCRBench v2 is the second-generation OCR/document-understanding benchmark suite with harder reading and parsing tasks. Recorded from the Bonsai 2 27B GGUF ca...
Overall (EN)
STEM & Reasoning
Kolibri card composite 'Overall (EN)': unweighted mean of the English category averages (Knowledge, Math, Agentic, Code, Instruction Following, Grounding/Hal...
Overall (DE)
STEM & Reasoning
Kolibri card composite 'Overall (DE)': unweighted mean of the German category averages (Knowledge, Math, Agentic Retrieval, Industry RAG). Evaluation via Ale...
GPQA Diamond (DE)
STEM & Reasoning
Aleph Alpha's official German translation of GPQA Diamond, evaluated with the AA eval-framework (same setup as EN GPQA Diamond on the Kolibri card).
Humanity's Last Exam (DE)
STEM & Reasoning
Aleph Alpha's official German translation of Humanity's Last Exam, evaluated with the AA eval-framework (same setup as EN HLE on the Kolibri card).
AIME 2025 (DE)
STEM & Reasoning
Aleph Alpha's official German translation of AIME 2025, evaluated with the AA eval-framework (pass@1, no tools; same setup as EN AIME 2025 on the Kolibri card).
AIME 2026 (DE)
STEM & Reasoning
Aleph Alpha's official German translation of AIME 2026, evaluated with the AA eval-framework (pass@1, no tools; same setup as EN AIME 2026 on the Kolibri card).
BFCL v4 (non-live AST)
Agentic
BFCL v4 (non-live AST) - function-calling AST check on non-live prompts; split of BFCL v4 as reported on the Kolibri card.
BFCL v4 (live)
Agentic
BFCL v4 (live) - split of BFCL v4 covering live/real user queries, as reported on the Kolibri card.
BFCL v4 (multi-turn)
Agentic
BFCL v4 (multi-turn) - multi-turn state/global evaluation split of BFCL v4 as reported on the Kolibri card.
BFCL v4 (memory)
Agentic
BFCL v4 (memory) - memory-category split of BFCL v4 as reported on the Kolibri card.
BFCL v4 (web search)
Agentic
BFCL v4 (web search) - web-search-category split of BFCL v4 as reported on the Kolibri card.
SQuAD (M/A Grounding Score)
Grounding Hallucination
SQuAD (M/A Grounding Score) - grounding-quality score on the SQuAD grounding setup (M/A = missing/answerable items) as run in the AA eval-framework on the Ko...
SQuAD (Utility Accuracy)
Grounding Hallucination
SQuAD (Utility Accuracy) - answer/utility accuracy on the SQuAD grounding setup as run in the AA eval-framework on the Kolibri card.
RGB Closed-Book
Grounding Hallucination
RGB Closed-Book - row of the RGB hallucination-detection suite (closed-book setting) as printed on the Kolibri card in AA eval-framework setup.
RGB Negative (Abstention)
Grounding Hallucination
RGB Negative (Abstention) - negative-query abstention rate of the RGB hallucination-detection suite as printed on the Kolibri card.
RGB Fact-Check (Error Correction)
Grounding Hallucination
RGB Fact-Check (Error Correction) - fact-check/error-correction row of the RGB hallucination-detection suite as printed on the Kolibri card.
FRAMES (<24k)
Agentic
FRAMES (prompts under 24k tokens) - factuality/retrieval/reasoning benchmark as run in the AA eval-framework; overlength prompts score 0 on long-context benc...
FRAMES (>24k)
Agentic
FRAMES (prompts over 24k tokens) - factuality/retrieval/reasoning benchmark as run in the AA eval-framework on the Kolibri card.
SealQA (no distractors, <24k)
Agentic
SealQA (no distractors, prompts under 24k tokens) - search-augmented fact-checking variant as run in the AA eval-framework on the Kolibri card.
SealQA (12 distractors, <24k)
Agentic
SealQA (12 distractor documents, prompts under 24k tokens) - search-augmented fact-checking variant as run in the AA eval-framework on the Kolibri card.
SealQA (no distractors, >24k)
Agentic
SealQA (no distractors, prompts over 24k tokens) - search-augmented fact-checking variant as run in the AA eval-framework on the Kolibri card.
SealQA (12 distractors, >24k)
Agentic
SealQA (12 distractor documents, prompts over 24k tokens) - search-augmented fact-checking variant as run in the AA eval-framework on the Kolibri card.
MuSiQue (EN)
Agentic
MuSiQue (EN) - agentic multi-hop retrieval QA (English) as run in the AA eval-framework on the Kolibri card; constitutes the Agentic Retrieval Average (EN) row.
Honeypot
Agentic
Honeypot - trap-resistance benchmark from the agentic retrieval suite as run in the AA eval-framework on the Kolibri card; higher is better.
Agentic Wiki QA (DE)
Agentic
Agentic Wiki QA (DE) - agentic QA over German Wikipedia as run in the AA eval-framework on the Kolibri card; constitutes the Agentic Retrieval Average (DE) row.
Semiconductors
Agentic
Semiconductors - domain of Aleph Alpha's Industry RAG suite (industry RAG over semiconductor documents, English) as run in the AA eval-framework on the Kolib...
German Public Sector
Agentic
German Public Sector - domain of Aleph Alpha's Industry RAG suite (industry RAG over German public-sector documents) as run in the AA eval-framework on the K...
Aerospace
Agentic
Aerospace - domain of Aleph Alpha's Industry RAG suite (industry RAG over aerospace documents) as run in the AA eval-framework on the Kolibri card.
Automotive Supplier
Agentic
Automotive Supplier - domain of Aleph Alpha's Industry RAG suite (industry RAG over automotive-supplier documents) as run in the AA eval-framework on the Kol...
Industrial Drive Technology
Agentic
Industrial Drive Technology - domain of Aleph Alpha's Industry RAG suite (industry RAG over industrial drive technology documents) as run in the AA eval-fram...
ToolSandbox
General Agents
Agentic tool-use benchmark with multi-turn tool conversations. As evaluated on the LFM2.5-24B-A2B model card: vLLM, temperature = 0, max output tokens = 1024...
LiveClaw
General Agents
Agentic benchmark reported on the LFM2.5-24B-A2B model card for live agent workflows. As evaluated: vLLM with temperature = 1.0 (vLLM default; the agent sets...
SWE-bench Verified (OpenCode)
Agentic Coding
SWE-bench Verified evaluated with the OpenCode agent harness. As evaluated on the LFM2.5-24B-A2B model card: vLLM, temperature = 0.6, max output tokens = 327...
HLE-Diamond (no tools)
STEM & Reasoning
Humanity's Last Exam (Diamond subset) evaluated without tool access, reported on the Muse Spark Open model card (xhigh reasoning effort, temperature 1.0, top...
WorldVQA (with tools)
Vision & Language
WorldVQA visual question answering evaluated with tools, as printed on the Muse Spark Open model card (xhigh reasoning effort). Distinct from RealWorldQA.