MRCR v2 256K (8-needle)

MRCR v2 (Multi-Round Co-reference Resolution) tests memory inside long conversations: several similar hidden facts ('needles', here eight) are buried in tens of thousands of tokens of chat history and the model must return the right one, including its context. The context length is part of the benchmark name (128K / 256K).

MRCR v2 256K (8-needle) measures reading and reasoning over very long inputs. In detail, it checks:

• distinguishing several very similar hidden facts
• reconstructing exact context of a conversation snippet
• faithful attention across 128K-256K tokens
• reliability when distractors deliberately look alike

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the MRCR v2 256K (8-needle) benchmark?

MRCR v2 (Multi-Round Co-reference Resolution) tests memory inside long conversations: several similar hidden facts ('needles', here eight) are buried in tens of thousands of tokens of chat history and the model must return the right one, including its context. The context length is part of the benchmark name (128K / 256K).

What does MRCR v2 256K (8-needle) test?

MRCR v2 256K (8-needle) mainly measures reading and reasoning over very long inputs. Concretely, it checks: distinguishing several very similar hidden facts; reconstructing exact context of a conversation snippet; faithful attention across 128K-256K tokens.

How is the MRCR v2 256K (8-needle) score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

Where do I get MRCR v2 256K (8-needle) data and scores?

The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/mrcr-v2-256k-8-needle/scores) or pull it directly from the Open LLM Radar API with any HTTP client.