AI Vuln-Discovery Benchmark Landscape

The vuln-discovery thesis long flagged the absence of a common cross-vendor benchmark as its largest measurement gap. Capability claims rested on vendor-reported numbers against private harnesses. As of mid-2026 that gap has narrowed but not closed. This page maps the benchmarks now available, what each measures, and what remains missing.

The benchmarks now form a layered stack

Each benchmark targets a different rung of the discovery-to-action pipeline. They are complementary, not redundant.

Benchmark	Author / origin	What it measures	Scale	Oracle
CyberGym	UC Berkeley (RDI)	Vulnerability reproduction — PoC from description + unpatched code	1,507 tasks / 188 projects	Crashes pre-patch, not post-patch
ExploitBench	CMU (Brumley + Lee)	Exploit development depth — 5-tier ladder to ACE	41 V8 CVEs / 16 flags	Where on the ladder the agent stalls
ExploitGym	Berkeley + MPI-SP + UCSB + ASU	Exploit development breadth — fraction of bugs solved	898 bugs (userspace, V8, kernel)	Code execution + dynamic flag, 2h window
SCONE-bench	Anthropic-supported	Smart-contract exploitation — value drained	(contracts)	Dollar value of funds extracted
CTI-REALM	Microsoft Research	Detection engineering — CTI → validated Sigma/KQL	25 / 50 tasks	Reward 0–1 over the full workflow

Offense deepens left-to-right (reproduce → develop → monetize); CTI-REALM is the defender-side mirror, turning intelligence into deployed detections. XBOW’s private StorageDrive web-exploit benchmark and MDASH’s harness-on-CyberGym result sit alongside as vendor-run surfaces.

One model dominates every public surface

Claude Mythos Preview leads every benchmark where it appears:¹

Benchmark	Mythos result	Next best
CyberGym L1	83.1%	GPT-5.5 81.8%
ExploitBench	ACE on 21/41 V8 CVEs (~half)	every other model ≤1 ACE
ExploitGym	157 intended / 226 captures	Opus 4.6: 15 / 36
SCONE-bench	$35M drained	next model $15M

The MDASH harness tops raw Mythos on CyberGym (88.45% vs 83.1%), a ~5-point “harness over model” delta.² Confidence is high for the ExploitBench, ExploitGym, and SCONE figures (primary Anthropic and arXiv sources) and low for the CyberGym leaderboard, which is self-reported.

What the gap actually is now

The gap narrowed from “no cross-vendor benchmark” to “no shared methodology + weak verification”. Cross-vendor benchmarks now exist — CyberGym ranks Anthropic, OpenAI, Zhipu, and Moonshot side by side; ExploitGym is a four-institution effort; CTI-REALM scores 16 labs’ models. The discovery-to-action pipeline is covered end to end. What is still missing is narrower and more tractable than the original framing:

1. No shared scale. Each benchmark uses its own targets, harness, and oracle, so a CyberGym percentage and an ExploitBench 21/41 cannot be placed on one axis.
2. Weak independent verification. The CyberGym leaderboard is self-reported (0 verified). ExploitBench/Gym/SCONE numbers come from the benchmark authors and Anthropic. No third party has reproduced the headline Mythos figures.
3. Contamination risk. As public corpora (CyberGym, OSS-Fuzz) become training targets, scores drift upward — the concern that motivated XBOW’s private-benchmark design.

Open questions

• A unifying meta-benchmark. Whether the community converges on one scale (or a normalized cross-benchmark index) is the open measurement question.
• CTI-REALM per-model table. Only the top reward range (0.624–0.685) is sourced; see CTI-REALM.
• Independent reproduction. No neutral party has re-run the Mythos numbers on any of these benchmarks.

See also

• Frontier AI for Vulnerability Discovery: the thesis this landscape serves.
• ExploitBench & ExploitGym · CyberGym · CTI-REALM: the component pages.

Footnotes

1. Exploit-development figures: Anthropic Frontier Red Team, exploit evals; ExploitBench, arXiv 2605.14153; ExploitGym, RDI Berkeley. The CyberGym L1 leaderboard is self-reported; see CyberGym. ↩

2. Microsoft Security Blog, Defense at AI speed (2026-05-12). See the page summary. ↩