RULER (1M)

RULER (1M tokens) checks what a model can really do with an ultra-long context of up to one million tokens: synthetic tasks such as finding hidden passphrases ('needles'), tracing variables or aggregating facts across the whole document. The score shows how sharply the model works near its advertised limit.

RULER (1M) measures reading and reasoning over very long inputs. In detail, it checks:

• finding information in a million tokens of input
• reliable retrieval of multiple hidden items simultaneously
• aggregation and multi-hop reasoning over very long text
• stable performance instead of collapsing near the limit

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the RULER (1M) benchmark?

RULER (1M tokens) checks what a model can really do with an ultra-long context of up to one million tokens: synthetic tasks such as finding hidden passphrases ('needles'), tracing variables or aggregating facts across the whole document. The score shows how sharply the model works near its advertised limit.

What does RULER (1M) test?

RULER (1M) mainly measures reading and reasoning over very long inputs. Concretely, it checks: finding information in a million tokens of input; reliable retrieval of multiple hidden items simultaneously; aggregation and multi-hop reasoning over very long text.

How is the RULER (1M) score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get RULER (1M) data and scores?

The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/ruler-1m/scores) or pull it directly from the Open LLM Radar API with any HTTP client.