CyberGym Benchmark
CyberGym is a large-scale public benchmark for AI-driven vulnerability analysis — a corpus of 1,507 real-world vulnerability reproduction tasks derived from historical vulnerabilities across 188 major OSS-Fuzz projects. Created by researchers at UC Berkeley (Sunblaze group; RDI — Center for Responsible Decentralized Intelligence): Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang, and Dawn Song. Paper: arXiv:2506.02548. Code: github.com/sunblaze-ucb/cybergym. Dataset: huggingface.co/datasets/sunblaze-ucb/cybergym. Blog: rdi.berkeley.edu/blog/cybergym. It is the load-bearing third-party evaluation surface for agentic vulnerability discovery systems, analogous in function to AgentDojo for prompt-injection robustness or MMLU for general-capability ranking.
Significance
CyberGym is presently the most-cited public leaderboard for AI-driven vulnerability reproduction. Its level-1 configuration (vulnerable source provided + high-level vulnerability description) makes it tractable for evaluation while remaining grounded in real CVEs. Higher difficulty levels — not yet covered in detail here — remove context to test blind discovery.
The benchmark’s role on the wiki:
- The first independently-verifiable comparison surface for agentic-AI-vuln-discovery claims by MDASH, future Anthropic Glasswing releases, and any subsequent vendor entries on the `ai-vuln-discovery` axis.
- Counterpart to AgentDojo (prompt-injection) and CLASP (capability-centric agent evaluation) in the four-quadrant red-team grid — CyberGym sits in the “real-world reproduction” slot.
Known Results
| System | Score | Source | Configuration |
|---|---|---|---|
| Microsoft MDASH | 88.45% | Microsoft, May 2026 | level 1 |
| Claude Mythos Preview (raw model) | 83.1% | Anthropic Glasswing, May 2026 | level 1 |
| Claude Opus 4.6 (raw model) | 66.6% | Anthropic Glasswing, May 2026 | level 1 |
Harness over model — the ~5-point delta. On Level 1, MDASH sits about 5 percentage points above the raw model (see the Known Results table above). The MDASH harness (multi-model ensemble + specialized agents + debate + dedup + automated PoC construction) adds roughly that delta over the raw model alone. This is the clearest quantitative measurement on the wiki of the “harness over model” architectural argument from both XBOW and Microsoft.
Cross-vendor leaderboard snapshot (2026-05-23)
A third-party aggregator (llm-stats) published a six-model Level-1 snapshot. Treat as low confidence — all six entries are self-reported, none independently verified.
| Rank | Model | Score |
|---|---|---|
| 1 | Claude Mythos Preview | 83.1% |
| 2 | GPT-5.5 (OpenAI) | 81.8% |
| 3 | Claude Opus 4.6 | 73.8% |
| 4 | Claude Opus 4.7 | 73.1% |
| 5 | GLM-5.1 (Zhipu AI) | 68.7% |
| 6 | Kimi K2.5 (Moonshot AI) | 41.3% |
Opus 4.6 score varies by source
Anthropic’s Glasswing material reports raw Opus 4.6 at 66.6%; the llm-stats snapshot lists 73.8%. Different harnesses or snapshot dates likely explain the ~7-point gap — a concrete instance of why cross-source CyberGym numbers are not directly comparable. The Mythos 83.1% figure is consistent across both.
Direct CyberGym sourcing — resolved 2026-05-15
Homepage, paper (arXiv 2506.02548), GitHub repo (
sunblaze-ucb/cybergym), Hugging Face dataset, and RDI Berkeley blog all sourced. UC Berkeley team attributed (Wang, Shi, He, Cai, Zhang, Dawn Song).
Evaluation Modes
Per the CyberGym homepage and the arXiv methodology paper:
- Vulnerability Reproduction (Level 1) — agents receive a vulnerability description and an unpatched codebase, then must generate a working proof-of-concept (PoC) exploit that triggers the target vulnerability. Success is verified when the PoC crashes the pre-patch version but does not crash the patched version. This is the mode the published vendor numbers in the Known Results table above target.
- Open-Ended Discovery — agents analyze latest codebases without prior vulnerability knowledge to identify new security flaws, mirroring real-world vulnerability discovery scenarios. This is the harder, blind-discovery mode and the bridge to the real-world impact numbers below.
Real-World Impact (as published)
CyberGym is not just a benchmark; the open-ended-discovery side has produced concrete novel findings:
- 35 zero-day vulnerabilities discovered.
- 17 incomplete patches identified across 15 projects — vulnerabilities that had been patched but where the patch did not fully close the issue.
- 10 unique zero-days that had persisted an average of 969 days (~2.65 years) in upstream code before discovery — a load-bearing data point for the Mythos-ready briefing’s argument that AI-driven vuln discovery surfaces decade-class latent bugs.
Limitations and Caveats
- Description quality matters: Microsoft’s failure analysis of MDASH’s remaining ~12% errors shows that 82% of wrong-area findings came from tasks with vague descriptions that also lacked function or file identifiers — description quality is a major factor in scan accuracy.
- Harness-format mismatch: agents occasionally constructed libFuzzer-style inputs when the benchmark task required honggfuzz format, producing otherwise-sound reproductions that fail on harness-format mismatch.
- OSS-Fuzz domain: CyberGym is biased toward C/C++ memory-safety bug classes typical of OSS-Fuzz; coverage of web vulns, prompt-injection, supply-chain, or AI-application classes is structurally limited.
- Public-benchmark contamination risk: as vendors target the leaderboard, model training data may absorb the corpus; the same concern that motivated XBOW’s StorageDrive private-benchmark design.
CMM / RA Maps-to
- CMM D7 (Observability & Detection) L4 — fits the four-quadrant red-team grid’s “real-world reproduction benchmark” slot. Should be cited alongside AgentDojo in CMM evidence checklists for D7 L4.
See Also
- MDASH — current leaderboard leader.
- Microsoft’s MDASH announcement — citing source.
- Frontier AI for Vulnerability Discovery — the wiki thesis CyberGym anchors as a benchmark surface.
- AgentDojo — sibling public benchmark, different bug class (prompt injection).
- Red Teaming for AI: Synthesis — wiki position on the four-quadrant grid.