ExploitBench & ExploitGym
ExploitBench and ExploitGym are two academic benchmarks for AI exploit-development capability, released in May 2026 and surfaced in Anthropic’s one-month Glasswing update. Both show Claude Mythos Preview as the strongest performer; Anthropic states it supported development of both. They extend the wiki’s benchmark coverage on the ai-vuln-discovery axis from vulnerability reproduction (the focus of CyberGym) toward exploit construction, the harder downstream capability of turning a found vulnerability into a working exploit.
The Two Benchmarks
| Benchmark | Authors | Scope | Oracle |
|---|---|---|---|
| ExploitBench | David Brumley + Seunghyun Lee (CMU); arXiv 2605.14153 | 41 patched V8 CVEs, 16 measurable flags on a 5-tier ladder | Where on the ladder does the agent stall? |
| ExploitGym | UC Berkeley + MPI-SP + UC Santa Barbara + Arizona State (Anthropic/OpenAI/Google contributors); RDI Berkeley | 898 vulnerabilities across OSS-Fuzz, V8, Linux kernel; 2-hour window, dynamic flag capture | What fraction of bugs does the agent solve? |
The two are complementary: ExploitBench reports depth, ExploitGym reports breadth of capture. (Confidence: high per primary arXiv + Anthropic Frontier Red Team sources.)
ExploitBench Capability Ladder
ExploitBench treats exploitation as a ladder rather than a binary, with 16 flags across five tiers (per Anthropic’s Frontier Red Team write-up):
| Tier | Capability |
|---|---|
| T5 | Coverage — reaching vulnerable code |
| T4 | Reproduction — proof-of-concept crash |
| T3 | Target primitives — sandbox-confined exploit |
| T2 | Generic primitives — sandbox escape |
| T1 | Full control — arbitrary code execution (ACE) |
Concrete Results (Mythos vs the field)
- ExploitBench (41 V8 CVEs): Mythos Preview achieves ACE on 21 of 41 CVEs (~half). Every other model managed one ACE or fewer; Opus 4.6 tops out at T3. Public models reach code and trigger crashes routinely, but ACE remains rare: only the private frontier crosses it.
- ExploitGym (898 bugs): Mythos Preview solves 157 tasks via the intended vulnerability and 226 total flag captures; Opus 4.6 manages 15 intended / 36 via alternative paths.
- SCONE-bench (smart contracts): a third Anthropic-supported benchmark. Mythos drains $35M of contract value against the next model’s $15M (~75% more).
Exploit development is being commoditized. Anthropic’s framing: “Mythos-level models will become widely available in the next 6–12 months.” The specialist knowledge required to turn a vulnerability into a working exploit is dropping — the capability that distinguished elite offensive researchers is moving into the commodity frontier.
Anthropic discusses what the benchmarks reveal about Mythos on its Frontier Red Team blog (ingested 2026-05-23).
Why It Matters
Begins to close the “no common third-party benchmark” gap. The vuln-discovery thesis flagged the absence of a common third-party benchmark as the largest measurement gap — vendor numbers dominated, and CyberGym was the closest candidate but covers reproduction rather than exploitation. ExploitBench and ExploitGym are independently published academic instruments that measure exploit development specifically. They do not yet make cross-vendor numbers directly comparable (each vendor still reports against different harnesses), but they add neutral, reproducible measurement infrastructure to the axis.
Open Questions
- Methodology unification. The benchmarks report against different harnesses, scales, and oracles. Each ranks multiple labs’ models, but scores are not yet inter-comparable on a single scale. The benchmark landscape maps how they layer.
- Independent verification. The CyberGym leaderboard is self-reported (0 verified); ExploitBench/ExploitGym scores come from the benchmark authors and Anthropic. No third party has reproduced the Mythos figures.
See Also
- CyberGym: the reproduction-focused benchmark already on the wiki.
- Glasswing initial update: the source surfacing these benchmarks.
- Frontier AI for Vulnerability Discovery: where the benchmark gap is tracked.