Chartography

Chartography is a benchmark for agents that perceive and act across visual interfaces. It was built to compare language models on a clearly defined set of tasks under the same grading rules, so scores are directly comparable between models.

Chartography measures agents that perceive and act across visual interfaces. In detail, it checks:

• interpreting what is on screen and deciding what to do next
• combining visual perception with tool or UI actions
• completing multi-step tasks that require looking and acting alternately
• recovering when a UI action does not produce the expected result

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the Chartography benchmark?

Chartography is a benchmark for agents that perceive and act across visual interfaces. It was built to compare language models on a clearly defined set of tasks under the same grading rules, so scores are directly comparable between models.

What does Chartography test?

Chartography mainly measures agents that perceive and act across visual interfaces. Concretely, it checks: interpreting what is on screen and deciding what to do next; combining visual perception with tool or UI actions; completing multi-step tasks that require looking and acting alternately.

How is the Chartography score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

Where do I get Chartography data and scores?

The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/chartography/scores) or pull it directly from the Open LLM Radar API with any HTTP client.