Malicious Cyber Request Refusal

Malicious Cyber Request Refusal is a benchmark for an aggregated index computed from several underlying benchmarks. The official description says: "Average refusal rate on malicious cyber-related prompts of JailbreakBench, StrongREJECT and AgentHarm (higher = safer)". Below we describe in simple words what benchmarks of this type generally examine.

Malicious Cyber Request Refusal measures an aggregated index computed from several underlying benchmarks. In detail, it checks:

• a weighted synthesis of many individual evaluations
• coverage (how many different skills the model performed on)
• recency (favouring newly released models)
• none of the weaknesses that a single narrow benchmark would hide

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

Official methodology note: Average refusal rate on malicious cyber-related prompts of JailbreakBench, StrongREJECT and AgentHarm (higher = safer). Blog: ML4 96.6%, the highest Mistral measures among open models.

What is the Malicious Cyber Request Refusal benchmark?

Malicious Cyber Request Refusal is a benchmark for an aggregated index computed from several underlying benchmarks. The official description says: 'Average refusal rate on malicious cyber-related prompts of JailbreakBench, StrongREJECT and AgentHarm (higher = safer)'. Below we describe in simple words what benchmarks of this type generally examine.

What does Malicious Cyber Request Refusal test?

Malicious Cyber Request Refusal mainly measures an aggregated index computed from several underlying benchmarks. Concretely, it checks: a weighted synthesis of many individual evaluations; coverage (how many different skills the model performed on); recency (favouring newly released models).

How is the Malicious Cyber Request Refusal score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get Malicious Cyber Request Refusal data and scores?

The official source is https://mistral.ai/news/mistral-large-4-oss/. The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/malicious-cyber-request-refusal/scores) or pull it directly from the Open LLM Radar API with any HTTP client.