GPQA Diamond

GPQA Diamond is the 'diamond' subset of the Google-Proof Q&A benchmark — the hardest and cleanest part. The questions are written and checked by PhD scientists and are designed so that a skilled non-expert with full internet access can still barely get them right. The model has to answer graduate-level science questions with four multiple-choice options.

GPQA Diamond measures hard reasoning in science, technology, engineering and math. In detail, it checks:

• expert-level physics, chemistry and biology knowledge
• real reasoning instead of memorised question patterns
• resistance to plausible distractors
• reliability without web access (the questions are not Googleable)

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the GPQA Diamond benchmark?

GPQA Diamond is the 'diamond' subset of the Google-Proof Q&A benchmark — the hardest and cleanest part. The questions are written and checked by PhD scientists and are designed so that a skilled non-expert with full internet access can still barely get them right. The model has to answer graduate-level science questions with four multiple-choice options.

What does GPQA Diamond test?

GPQA Diamond mainly measures hard reasoning in science, technology, engineering and math. Concretely, it checks: expert-level physics, chemistry and biology knowledge; real reasoning instead of memorised question patterns; resistance to plausible distractors.

How is the GPQA Diamond score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get GPQA Diamond data and scores?

The official source is https://huggingface.co/datasets/Idavidrein/gpqa. The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/gpqa-diamond/scores) or pull it directly from the Open LLM Radar API with any HTTP client.