MGSM

MGSM (Multilingual Grade School Math) is GSM8K translated into about ten other languages. It shows whether a model's math ability survives when the problem is written in Spanish, German, Bengali, Swahili and so on — a good test of language transfer.

MGSM measures performance in languages other than english. In detail, it checks:

• same math quality across many languages
• understanding of non-English problem statements
• reasoning transfer from English to other languages
• fair treatment of low-resource languages

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the MGSM benchmark?

MGSM (Multilingual Grade School Math) is GSM8K translated into about ten other languages. It shows whether a model's math ability survives when the problem is written in Spanish, German, Bengali, Swahili and so on — a good test of language transfer.

What does MGSM test?

MGSM mainly measures performance in languages other than English. Concretely, it checks: same math quality across many languages; understanding of non-English problem statements; reasoning transfer from English to other languages.

How is the MGSM score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get MGSM data and scores?

The official source is https://arxiv.org/abs/2210.03057. The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/mgsm/scores) or pull it directly from the Open LLM Radar API with any HTTP client.