VideoMME (w/o sub.)

VideoMME evaluates video understanding: the model watches videos of up to an hour and answers multiple-choice questions about event order, details and cause-and-effect relationships. The variants run with and without subtitles, which shows how well a model reads the actual video track.

VideoMME (w/o sub.) measures understanding videos instead of static images. In detail, it checks:

• following events, people and their order over time
• answering about details that only appear in the video
• short, medium and long videos in one test
• understanding without subtitle support (or with, per variant)

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the VideoMME (w/o sub.) benchmark?

VideoMME evaluates video understanding: the model watches videos of up to an hour and answers multiple-choice questions about event order, details and cause-and-effect relationships. The variants run with and without subtitles, which shows how well a model reads the actual video track.

What does VideoMME (w/o sub.) test?

VideoMME (w/o sub.) mainly measures understanding videos instead of static images. Concretely, it checks: following events, people and their order over time; answering about details that only appear in the video; short, medium and long videos in one test.

How is the VideoMME (w/o sub.) score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get VideoMME (w/o sub.) data and scores?

The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/videomme-wo-sub/scores) or pull it directly from the Open LLM Radar API with any HTTP client.