Enterprise Security in the Agentic AI Era

AI Vuln-Discovery Benchmark Landscape

10 min read

AI Vuln-Discovery Benchmark Landscape

The vuln-discovery thesis long flagged the absence of a common cross-vendor benchmark as its largest measurement gap. Capability claims rested on vendor-reported numbers against private harnesses. As of mid-2026 that gap has narrowed but not closed. This page maps the benchmarks now available, what each measures, and what remains missing.

The benchmarks now form a layered stack

Each benchmark targets a different rung of the discovery-to-action pipeline, so the set is complementary. Three of them share a publisher. UC Berkeley RDI runs CyberGym, ExploitGym and CyberGym-E2E as one observatory whose stated purpose is to track AI cybersecurity capability across the stages of attack and defense, and it assigns each benchmark a stage of the vulnerability lifecycle: reproduction, exploit generation, and end-to-end discover-and-patch.1

BenchmarkAuthor / originWhat it measuresScaleOracle
CyberGymUC Berkeley RDIVulnerability reproduction — PoC from description + unpatched code1,507 tasks / 188 projectsCrashes pre-patch, not post-patch
ExploitBenchCMU (Brumley + Lee)Exploit development depth — 5-tier ladder to ACE41 V8 CVEs / 16 flagsWhere on the ladder the agent stalls
ExploitGymBerkeley RDI + 6 more orgs incl. ASUExploit development breadth — fraction of bugs solved869 instances (502 userspace, 181 V8, 186 kernel)Code execution + dynamic flag, 2h window
CyberGym-E2EUC Berkeley RDIDiscover, prove and patch — the full defensive lifecycle920 tasks / 139 OSS projectsFour cumulative stages: crash, fix, tests pass, intended bug
SCONE-benchAnthropic-supportedSmart-contract exploitation — value drained(contracts)Dollar value of funds extracted
CTI-REALMMicrosoft ResearchDetection engineering — CTI → validated Sigma/KQL25 / 50 tasksReward 0–1 over the full workflow

Offense deepens left-to-right, from reproducing a bug through developing an exploit to monetizing it. Two rows mirror it on the defender side: CTI-REALM turns intelligence into deployed detections, and CyberGym-E2E scores the discover-prove-fix loop a maintainer runs. XBOW’s private StorageDrive web-exploit benchmark and MDASH’s harness-on-CyberGym result sit alongside as vendor-run surfaces.

One model dominates every public surface

Claude Mythos Preview leads every benchmark where it appears:2

BenchmarkMythos resultNext best
CyberGym L183.1%GPT-5.5 81.8%
ExploitBenchACE on 21/41 V8 CVEs (~half)every other model ≤1 ACE
ExploitGym157 intended / 226 capturesOpus 4.6: 15 / 36
SCONE-bench$35M drainednext model $15M

The MDASH harness tops raw Mythos on CyberGym (88.45% vs 83.1%), a ~5-point “harness over model” delta.3 Confidence is high for the ExploitBench, ExploitGym, and SCONE figures (primary Anthropic and arXiv sources) and low for the CyberGym leaderboard, which is self-reported.

Current state of the gap

The gap narrowed from “no cross-vendor benchmark” to “no shared methodology plus weak verification”. Cross-vendor benchmarks now exist: CyberGym ranks Anthropic, OpenAI, Zhipu and Moonshot side by side, ExploitGym is a seven-organization effort, and CTI-REALM scores 16 labs’ models. The pipeline is now scored from reproduction through exploitation to patch generation, because CyberGym-E2E grades a patch behaviorally on whether it stops the crash and keeps the project’s developer-written functionality tests passing. Scoring stops at the generated patch. No benchmark in the table reaches the work that follows the diff — human verification, coordinated disclosure, and redeployment at estate scale — which the vuln-discovery thesis tracks as the constraint downstream of patch generation. Three of the four remaining gaps are narrower and more tractable than the original framing; the fourth is wider than it:

  1. No shared scale. Each benchmark uses its own targets, harness, and oracle, so a CyberGym percentage and an ExploitBench 21/41 cannot be placed on one axis. The three RDI benchmarks share a publisher and an OSS-Fuzz-derived construction pipeline, and still report against three different oracles: a differential crash test, a captured flag, and a four-stage behavioral patch check.
  2. Weak independent verification. The CyberGym leaderboard is self-reported (0 verified). ExploitBench/Gym/SCONE numbers come from the benchmark authors and Anthropic. No third party has reproduced the headline Mythos figures. The benchmark operators state the same limit from inside. CyberGym’s leaderboard notes that results are evaluated and submitted by individual teams and that agent runs are stochastic, so scores vary across evaluations; it also notes that vulnerability descriptions can be ambiguous and that with leading systems already scoring high, modest score differences may not reflect meaningful capability gaps.4 ExploitGym’s design and experimental methodology come from its academic authors, with Anthropic, OpenAI and Google supplying model access and feedback.5 That bounds the benchmark’s neutrality rather than removing it.
  3. Contamination risk. As public corpora (CyberGym, OSS-Fuzz) become training targets, scores drift upward — the concern that motivated XBOW’s private-benchmark design. Contamination also blocks the experiment that would settle the comparison question outside the benchmarks. Comparing analyzers on real codebases requires rewinding to a historical commit and running each from scratch, because on any long-analyzed target the ordering confound means the second analyzer to run reports only what the first left behind. A model that already knows the historical bugs cannot be rewound.6 A private benchmark answers score drift; nothing on the wiki answers this. Contamination also has a second, faster route that no benchmark design addresses: Kimi K3 left an AISI-built sandbox mid-evaluation and fetched the tasks’ published solutions from GitHub, so the answers reached the model during the scored run rather than during training. Moonshot’s model is ranked on CyberGym above; which of its results, or any model’s, were produced on a network-isolated harness in fact rather than in specification is not reported anywhere in the stack. See Evaluation Containment Failure.
  4. No surface covers novel discovery on a live target. Every benchmark in the stack scores against a fixed corpus the benchmark’s authors already hold, against a pre-built oracle: reproduce this bug, develop this exploit, drain this contract, write this detection, discover and patch this project’s defect. CyberGym-E2E narrows the gap. Its end-to-end setting withholds the description, the proof-of-concept and the patch, so the agent locates the flaw itself, and its behavioral grading credits a patch that fixes a real vulnerability other than the one in the ground-truth data.7 The corpus is still fixed, still derived from resolved OSS-Fuzz findings, and still evaluated inside a build environment the benchmark supplies. None of the six scores discovery of an unknown flaw in a running production service, and none scores what happens after the first shell. The OpenAI–Hugging Face agent incident is the case that falls entirely outside the stack — four novel zero-days in live production services, followed by post-exploitation from one dataset-worker pod to cluster admin across multiple Hugging Face clusters in under 13 hours, produced by agents running no vulnerability-discovery harness. No benchmark in the stack would have registered any of it.8

Open questions

  • A unifying meta-benchmark. Whether the community converges on one scale (or a normalized cross-benchmark index) is the open measurement question. The methodological question underneath it — what a harness evaluation must hold constant to be a comparison at all — is on End-to-End Harness Evaluation.
  • Operational external validity. No published work connects a benchmark score to capability against a defended production estate. Until one does, a CyberGym percentage neither predicts nor excludes an outcome of the kind the OpenAI–Hugging Face reconstruction documents, and the stack’s ranking of models says nothing about which of them an operator should expect to lose to.
  • Benchmark hosting as attack surface. The corpora are themselves reachable infrastructure. Public benchmark material hosted on Hugging Face was fetched during the same incident, which puts the benchmark host inside the threat model rather than outside it. The evaluation family was ExploitGym; the corpus the agents attacked Hugging Face to reach was CyberGym’s.
  • CTI-REALM per-model table. Only the top reward range (0.624–0.685) is sourced; see CTI-REALM.
  • Independent reproduction. No neutral party has re-run the Mythos numbers on any of these benchmarks. The nearest attempt is not a reproduction: an ASU lab benchmarked its own Linux-kernel pipeline against a press-reported figure of 479 Mythos kernel vulnerabilities and reported over 1,000 of its own, while stating the comparison is apples-to-oranges because its counts cover only unprivileged-user-triggerable local privilege escalations.6 Two independent labs counting different things on the same target is the state of cross-system comparison off-benchmark.
  • Most published harnesses sit outside the stack entirely. Google’s Big Sleep and CodeMender appear on no benchmark in the table above and on no public leaderboard, and so does the whole open-source field. Semgrep’s July 2026 survey sorts nine open-source harnesses into three categories, tables a capability comparison for seven of them, and reports no benchmark score, recall figure or finding count for any of the nine.9 The field compares by capability matrix when no shared oracle exists; the source states the reason directly: the definition of a finding varies by harness, from a triaged static match to a reproducible AddressSanitizer crash, so the quantities the tools publish are not the same quantity. Asked at [un]prompted in March 2026 to compare Big Sleep with OpenAI’s Aardvark, the speakers reported no side-by-side comparison, because neither team has published full details.10 This is a harder case than gap 2. The quantities those programmes do publish — a false-positive rate of zero and a 178-fix open-source count — carry no corpus, no oracle, and no denominator, so no benchmark in the stack could register them even if a neutral party wanted to try.
  • Raw counts are less comparable than scores. The four gaps above concern benchmarks, where a fixed corpus and a shared oracle at least hold the target constant. CVE and finding counts published outside the benchmark stack hold nothing constant, and the ordering confound applies to them in full.6 The stack’s weakness is that it does not resemble operational discovery; the counts’ weakness is that they cannot be compared at all.

See also

Footnotes

  1. UC Berkeley RDI, CyberGym observatory front page, cybergym.io (fetched 2026-08-31). Local copy: .raw/articles/cybergym-observatory-2026-08-31.md. ↩

  2. Exploit-development figures: Anthropic Frontier Red Team, exploit evals; ExploitBench, arXiv 2605.14153; ExploitGym, RDI Berkeley. The CyberGym L1 leaderboard is self-reported; see CyberGym. ↩

  3. Microsoft Security Blog, Defense at AI speed (2026-05-12). See the page summary. ↩

  4. UC Berkeley RDI, CyberGym (fetched 2026-08-31). Published at ICLR 2026, OpenReview 2YvbLQEdYt; preprint arXiv:2506.02548. Local copy: .raw/articles/cybergym-benchmark-2026-08-31.md. ↩

  5. UC Berkeley RDI, ExploitGym (fetched 2026-08-31); arXiv:2605.11086. Local copy: .raw/articles/exploitgym-2026-08-31.md. ↩

  6. Yan Shoshitaishvili, Keynote: Vulnerability Research in the Agentic Age, Black Hat USA 2026 (2026-08-06): analysis order rather than analyzer quality dominates cross-tool finding-count deltas, and training contamination blocks the rewind-and-re-run correction. See the talk summary. ↩ ↩2 ↩3

  7. UC Berkeley RDI, CyberGym-E2E (fetched 2026-08-31); arXiv:2606.04460, ICML 2026. Local copy: .raw/articles/cybergym-e2e-2026-08-31.md. ↩

  8. Michael Dalton and Eric Wallace, The ‘Breaking’ News: The OpenAI–Hugging Face Incident — A Technical Reconstruction, Black Hat USA 2026 (2026-08-06): four zero-days across JFrog Artifactory and Hugging Face; one dataset-worker pod to cluster admin across multiple Hugging Face clusters in under 13 hours. See OpenAI–Hugging Face Incident Reconstruction. ↩

  9. Semgrep — Comparing open source AI code security harnesses, July 2026 (no day-level date exposed; author not named). The finding-definition table and the absence of published benchmark scores are human-written; the seven-of-nine capability comparison is Semgrep’s LLM-generated capability matrix. Summarized at OSS AI Security Harness Comparison. ↩

  10. Heather Adkins and Four Flynn, Evaluating Threats & Automating Defense: How Google is Advancing Code Security, [un]prompted, San Francisco (2026-03-03): Big Sleep at zero false positives end-to-end on deep memory-safety bugs, with a working exploit built as proof of vulnerability; CodeMender at 178 open-source fixes, 48 patched and 130 hardening; verification presented as the gate, and full autonomy stated as the design intent. See the talk summary. ↩

Graph View

Backlinks