BigBench Extra Hard

BigBench Extra Hard (BBEH) takes selected BIG-Bench reasoning tasks and rebuilds them to be substantially harder, so that even strong models can no longer separate the tasks with heuristics. It measures how robust general reasoning really is.

BigBench Extra Hard measures hard reasoning in science, technology, engineering and math. In detail, it checks:

• logical and linguistic reasoning tasks
• resistance to previously learned 'task templates'
• multi-step deductions where small errors cascade
• generalisation, not memorised solutions

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the BigBench Extra Hard benchmark?

BigBench Extra Hard (BBEH) takes selected BIG-Bench reasoning tasks and rebuilds them to be substantially harder, so that even strong models can no longer separate the tasks with heuristics. It measures how robust general reasoning really is.

What does BigBench Extra Hard test?

BigBench Extra Hard mainly measures hard reasoning in science, technology, engineering and math. Concretely, it checks: logical and linguistic reasoning tasks; resistance to previously learned 'task templates'; multi-step deductions where small errors cascade.

How is the BigBench Extra Hard score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get BigBench Extra Hard data and scores?

The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/bigbench-extra-hard/scores) or pull it directly from the Open LLM Radar API with any HTTP client.