MMLU-Redux

MMLU-Redux is a cleaned-up subset of MMLU: ambiguous or obviously wrong questions were hand-flagged and removed, and remaining items were re-checked. The result measures the same knowledge as MMLU, but without being distorted by erroneous test items.

MMLU-Redux measures broad factual and academic knowledge. In detail, it checks:

• broad knowledge on verified question data
• removal of mislabelled or ambiguous items
• more trustworthy per-field numbers than raw MMLU
• stable comparability between models

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the MMLU-Redux benchmark?

MMLU-Redux is a cleaned-up subset of MMLU: ambiguous or obviously wrong questions were hand-flagged and removed, and remaining items were re-checked. The result measures the same knowledge as MMLU, but without being distorted by erroneous test items.

What does MMLU-Redux test?

MMLU-Redux mainly measures broad factual and academic knowledge. Concretely, it checks: broad knowledge on verified question data; removal of mislabelled or ambiguous items; more trustworthy per-field numbers than raw MMLU.

How is the MMLU-Redux score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get MMLU-Redux data and scores?

The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/mmlu-redux/scores) or pull it directly from the Open LLM Radar API with any HTTP client.