NL2Repo

NL2Repo asks for something very hard: generate a complete, working software repository from a natural-language description. The evaluated model typically runs inside an coding agent harness (for example Claude Code) and has many turns to create, run and fix its own code.

NL2Repo measures how well a model works as an autonomous coding agent. In detail, it checks:

• turning a plain-language spec into a project structure
• writing complete code files instead of fragments
• making the generated repo actually build and run
• long autonomous sessions with hundreds of tool calls

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the NL2Repo benchmark?

NL2Repo asks for something very hard: generate a complete, working software repository from a natural-language description. The evaluated model typically runs inside an coding agent harness (for example Claude Code) and has many turns to create, run and fix its own code.

What does NL2Repo test?

NL2Repo mainly measures how well a model works as an autonomous coding agent. Concretely, it checks: turning a plain-language spec into a project structure; writing complete code files instead of fragments; making the generated repo actually build and run.

How is the NL2Repo score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get NL2Repo data and scores?

The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/nl2repo/scores) or pull it directly from the Open LLM Radar API with any HTTP client.