ExploitBench & ExploitGym
ExploitBench and ExploitGym are two academic benchmarks for AI exploit-development capability, released in May 2026 and surfaced in Anthropic’s one-month Glasswing update. Both show Claude Mythos Preview as the strongest performer; Anthropic states it supported development of both. They extend the wiki’s AI vulnerability-discovery benchmark coverage from vulnerability reproduction (the focus of CyberGym) toward exploit construction, the harder downstream capability of turning a found vulnerability into a working exploit.
The Two Benchmarks
| Benchmark | Authors | Scope | Oracle |
|---|---|---|---|
| ExploitBench | David Brumley + Seunghyun Lee (CMU); arXiv 2605.14153 | 41 patched V8 CVEs, 16 measurable flags on a 5-tier ladder | Where on the ladder does the agent stall? |
| ExploitGym | UC Berkeley RDI + 6 more orgs incl. Arizona State1 | 869 instances across userspace, V8 and the Linux kernel; 2-hour default window, dynamic flag capture | What fraction of bugs does the agent solve? |
The two are complementary: ExploitBench reports depth, ExploitGym reports breadth of capture. (Confidence: high per primary arXiv + Anthropic Frontier Red Team sources.)
Yan Shoshitaishvili is a named co-author of ExploitGym, and the benchmark’s attribution paragraph lists Arizona State University among seven contributing organizations: UC Berkeley, the Max Planck Institute for Security and Privacy, UC Santa Barbara, Arizona State University, Anthropic, OpenAI and Google.1 The overlap is one person on one benchmark. Shoshitaishvili appears on no CyberGym or CyberGym-E2E author list, the attribution names the university rather than the SEFCOM lab, and the Linux-kernel and OpenHarmony pipelines his group reports are not benchmarked here. ExploitGym’s design and experimental methodology are the academic authors’; Anthropic, OpenAI and Google supplied model access and feedback, which bounds the benchmark’s neutrality rather than removing it.1
ExploitBench Capability Ladder
ExploitBench treats exploitation as a ladder rather than a binary, with 16 flags across five tiers (per Anthropic’s Frontier Red Team write-up):
| Tier | Capability |
|---|---|
| T5 | Coverage — reaching vulnerable code |
| T4 | Reproduction — proof-of-concept crash |
| T3 | Target primitives — sandbox-confined exploit |
| T2 | Generic primitives — sandbox escape |
| T1 | Full control — arbitrary code execution (ACE) |
Concrete Results (Mythos vs the field)
- ExploitBench (41 V8 CVEs): Mythos Preview achieves ACE on 21 of 41 CVEs (~half). Every other model managed one ACE or fewer; Opus 4.6 tops out at T3. Public models reach code and trigger crashes routinely, but ACE remains rare: only the private frontier crosses it.
- ExploitGym (869 instances: 502 userspace, 181 V8, 186 Linux kernel): Mythos Preview captures 226 flags, 157 of them through the intended vulnerability. GPT-5.5 captures 210 flags, 120 through the intended vulnerability.1 Opus 4.6 manages 15 intended and 36 via alternative paths.
- SCONE-bench (smart contracts): a third Anthropic-supported benchmark. Mythos drains $35M of contract value against the next model’s $15M (~75% more).
What these numbers say about how exploits are built and how the claim is graded — the ensemble finding, the time-budget curve, the mitigation results, and the gap between total captures and intended-vulnerability solves — is on Autonomous Exploit Generation. This page describes the instruments.
Exploit development is being commoditized. Anthropic’s framing: “Mythos-level models will become widely available in the next 6–12 months.” The specialist knowledge required to turn a vulnerability into a working exploit is dropping — the capability that distinguished elite offensive researchers is moving into the commodity frontier.
Anthropic discusses what the benchmarks reveal about Mythos on its Frontier Red Team blog (ingested 2026-05-23).
Significance
Begins to close the “no common third-party benchmark” gap. The vuln-discovery thesis flagged the absence of a common third-party benchmark as the largest measurement gap — vendor numbers dominated, and CyberGym was the closest candidate but covers reproduction rather than exploitation. ExploitBench and ExploitGym are independently published academic instruments that measure exploit development specifically. They do not yet make cross-vendor numbers directly comparable (each vendor still reports against different harnesses), but they add neutral, reproducible measurement infrastructure to the axis.
Open Questions
- Methodology unification. The benchmarks report against different harnesses, scales, and oracles. Each ranks multiple labs’ models, but scores are not yet inter-comparable on a single scale. The benchmark landscape maps how they layer.
- Independent verification. The CyberGym leaderboard is self-reported (0 verified); ExploitBench/ExploitGym scores come from the benchmark authors and Anthropic. No third party has reproduced the Mythos figures.
See Also
- End-to-End Harness Evaluation — ExploitGym’s flag-capture oracle and agent-as-a-judge attribution supply the exploitation row of that page’s path-attribution table.
- CyberGym: the reproduction-focused benchmark already on the wiki.
- Glasswing initial update: the source surfacing these benchmarks.
- Frontier AI for Vulnerability Discovery: where the benchmark gap is tracked.
- CyberGym-E2E: the third benchmark from the same publisher, scoring discovery and patching around the exploitation stage these two measure.
Sources
Footnotes
-
UC Berkeley RDI, ExploitGym (fetched 2026-08-31); arXiv:2605.11086. Local copy:
.raw/articles/exploitgym-2026-08-31.md. ↩ ↩2 ↩3 ↩4