ECLeKTic

ECLeKTIC is a multilingual knowledge benchmark: the same obscure knowledge items are asked in twelve languages. The result shows how unequally knowledge is distributed within a model across languages — the gap between best and worst language is telling.

ECLeKTic measures performance in languages other than english. In detail, it checks:

• closed-book knowledge in twelve different languages
• equality of knowledge across languages
• cultural and linguistic long-tail knowledge
• no silent fallback to English

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the ECLeKTic benchmark?

ECLeKTIC is a multilingual knowledge benchmark: the same obscure knowledge items are asked in twelve languages. The result shows how unequally knowledge is distributed within a model across languages — the gap between best and worst language is telling.

What does ECLeKTic test?

ECLeKTic mainly measures performance in languages other than English. Concretely, it checks: closed-book knowledge in twelve different languages; equality of knowledge across languages; cultural and linguistic long-tail knowledge.

How is the ECLeKTic score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get ECLeKTic data and scores?

The official source is https://arxiv.org/abs/2502.21228. The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/eclektic/scores) or pull it directly from the Open LLM Radar API with any HTTP client.