SciCode

SciCode tests whether a model can write scientific research code: programming problems are derived from real research papers (physics, chemistry, materials science and more) and the code must reproduce the paper's computational steps correctly. This is much closer to a scientist's job than website hacking.

SciCode measures the abilities the benchmark was built to measure. In detail, it checks:

• translating scientific methods into working code
• correct numerical results against reference implementations
• understanding research-level problem descriptions
• compositional solutions across several sub-steps

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the SciCode benchmark?

SciCode tests whether a model can write scientific research code: programming problems are derived from real research papers (physics, chemistry, materials science and more) and the code must reproduce the paper's computational steps correctly. This is much closer to a scientist's job than website hacking.

What does SciCode test?

SciCode mainly measures the abilities the benchmark was built to measure. Concretely, it checks: translating scientific methods into working code; correct numerical results against reference implementations; understanding research-level problem descriptions.

How is the SciCode score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

Where do I get SciCode data and scores?

The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/scicode/scores) or pull it directly from the Open LLM Radar API with any HTTP client.