HellaSwag

HellaSwag is a commonsense benchmark: for each situation the model has to pick the most sensible continuation from four options. The wrong alternatives were generated adversarially so they look plausible, but a human solves almost everything correctly (≈95%).

HellaSwag measures general reasoning and comprehension. In detail, it checks:

• commonsense judgement of everyday situations
• plausible-but-wrong distractors (adversarially generated)
• longer sentence understanding, not single-word decisions
• situational coherence over several sentences

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the HellaSwag benchmark?

HellaSwag is a commonsense benchmark: for each situation the model has to pick the most sensible continuation from four options. The wrong alternatives were generated adversarially so they look plausible, but a human solves almost everything correctly (≈95%).

What does HellaSwag test?

HellaSwag mainly measures general reasoning and comprehension. Concretely, it checks: commonsense judgement of everyday situations; plausible-but-wrong distractors (adversarially generated); longer sentence understanding, not single-word decisions.

How is the HellaSwag score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get HellaSwag data and scores?

The official source is https://arxiv.org/abs/1905.07830. The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/hellaswag/scores) or pull it directly from the Open LLM Radar API with any HTTP client.