MMLU

MMLU (Massive Multitask Language Understanding) is the classic general-knowledge benchmark: multiple-choice questions from 57 subjects — from physics and law to US history. For years it has shown how broad a model's knowledge is; the variants MMLU-Pro and MMLU-Redux today measure the same thing more cleanly.

MMLU measures broad factual and academic knowledge. In detail, it checks:

• breadth of knowledge over 57 different subjects
• multiple-choice answering accuracy
• academic content from school to professional level
• reproducibility, as one of the longest-running standard benchmarks

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the MMLU benchmark?

MMLU (Massive Multitask Language Understanding) is the classic general-knowledge benchmark: multiple-choice questions from 57 subjects — from physics and law to US history. For years it has shown how broad a model's knowledge is; the variants MMLU-Pro and MMLU-Redux today measure the same thing more cleanly.

What does MMLU test?

MMLU mainly measures broad factual and academic knowledge. Concretely, it checks: breadth of knowledge over 57 different subjects; multiple-choice answering accuracy; academic content from school to professional level.

How is the MMLU score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get MMLU data and scores?

The official source is https://huggingface.co/datasets/cais/mmlu. The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/mmlu/scores) or pull it directly from the Open LLM Radar API with any HTTP client.