CyberSecEval
Sources: Purple Llama / CyberSecEval (repo)
Meta’s open benchmark suite (MIT-licensed, shipped in Purple Llama) for measuring the cybersecurity risk a language model carries: both the insecure code it writes and the offensive capability it can supply to an attacker. It is the measurement counterpart to Purple Llama’s defensive guardrails, quantifying the ai-in-sec-offense risk a defender needs before tuning a sec-of-ai control.
The three generations
| Version | Adds |
|---|---|
| CyberSecEval v1 | First benchmarks grounded in CWE and MITRE ATT&CK; scores insecure-code suggestions and compliance with malicious requests |
| CyberSecEval 2 | Code-interpreter abuse, offensive-capability tests, prompt-injection susceptibility; Hugging Face leaderboard |
| CyberSecEval 3 | Visual prompt injection, spear-phishing assessment, autonomous offensive-operations tests |
How the wiki uses it
CyberSecEval is the recurring external yardstick behind two existing pages. Google Project Zero’s Naptime agent reached state-of-the-art on CyberSecEval2 before evolving into Big Sleep; the benchmark is how that line of frontier vulnerability-discovery work measured its offensive uplift. CyberSecEval3 supplied the manually labeled insecure-code completions (50 per language) used to validate CodeShield in the LlamaFirewall paper.
The suite sits on the offense side of the same purple-team split that produces LlamaFirewall: CyberSecEval measures the capability, the guardrails contain it. For the CMM, it is a candidate evidence source wherever a domain calls for quantified model-capability risk rather than control presence.
Distinction from AgentDojo
AgentDojo is a peer-reviewed prompt-injection benchmark for tool-using agents; CyberSecEval is a model-capability benchmark (insecure code, offensive uplift, phishing). They answer different questions. AgentDojo asks “does this defense stop the injection”; CyberSecEval asks “how dangerous is this model’s raw capability.” The LlamaFirewall evaluation draws on both.
See also
- Purple Llama — the parent project
- LlamaFirewall paper — uses CyberSecEval3 for the CodeShield evaluation
- Big Sleep — benchmarked against CyberSecEval2
- AgentDojo — the prompt-injection-specific independent benchmark