BIG-Bench Hard

BIG-Bench Hard (BBH) is a subset of 23 challenging tasks from the crowd-sourced BIG-Bench collection on which earlier language models still clearly trailed the human average. Today BBH is one of the most established reasoning baselines.

BIG-Bench Hard measures general reasoning and comprehension. In detail, it checks:

• 23 very different reasoning task types
• logical, linguistic and spatial reasoning
• multi-step thinking with chain-of-thought
• comparability as an established baseline

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the BIG-Bench Hard benchmark?

BIG-Bench Hard (BBH) is a subset of 23 challenging tasks from the crowd-sourced BIG-Bench collection on which earlier language models still clearly trailed the human average. Today BBH is one of the most established reasoning baselines.

What does BIG-Bench Hard test?

BIG-Bench Hard mainly measures general reasoning and comprehension. Concretely, it checks: 23 very different reasoning task types; logical, linguistic and spatial reasoning; multi-step thinking with chain-of-thought.

How is the BIG-Bench Hard score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get BIG-Bench Hard data and scores?

The official source is https://paperswithcode.com/dataset/bbh. The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/big-bench-hard/scores) or pull it directly from the Open LLM Radar API with any HTTP client.