Agentic Vulnerability Discovery

Agentic vulnerability discovery directs an LLM agent to locate a defect in a codebase and produce an input that proves the defect exists. This page states how that work is done and how the resulting claim is validated. Frontier AI for Vulnerability Discovery carries where the field stands and who ships the capability; CyberGym Benchmark describes one instrument in detail.

The evidence below comes from CyberGym, published by UC Berkeley RDI as the reproduction stage of a three-benchmark observatory over the vulnerability lifecycle.¹ CyberGym holds 1,507 instances across 188 software projects, each derived from a vulnerability that OSS-Fuzz, Google’s continuous fuzzing campaign, had already found and that the project had already patched.² Every figure on this page comes from that observatory, and the limits section states what the corpus can and cannot support.

Method

The working loop

An agent reproducing a vulnerability retrieves, constructs, and mutates under execution feedback. It browses the source tree, runs targeted keyword search to reach candidate code, reads the surrounding parser or handler, assembles a candidate input from what it read, executes it, and mutates that input against the crash output until the target crash appears.² The loop samples far fewer inputs than a fuzzer and reasons over far less of the program than static analysis. Its cost accrues in close code reading and in execution round trips, so the levers that move it are the context the agent receives and the time it is allowed to run.

The loop runs in two regimes, and the levers below divide between them. In the reproduction regime a report already names the defect — a fuzzer crash, a filed bug, an advisory — and the work is to reach it; input richness and reasoning budget are the levers there. In the open-ended regime nothing names the defect, the agent starts from current code alone, and the levers are breadth, run-length policy and the complexity of input the agent can synthesize. The two regimes reward different first investments, so a pipeline settles which regime it is in before it settles a budget.

Input richness

The context handed to the agent moves the score more than the choice of model does. CyberGym grades the same corpus at four input levels. At Level 0 the agent receives no text description of the target vulnerability and reproduces 3.5% of instances. Level 1 supplies a description and the unpatched codebase, and is the level the leaderboard scores. Level 2 adds the stack trace and Level 3 adds the ground-truth patch; the publisher reports that both greatly increase the reproduction rate over Level 1.²

A pipeline working from an existing report should therefore spend its first engineering effort on manufacturing a description, a stack trace and a crash log for each target, because that context moves the rate further than a model upgrade does. An open-ended pipeline has no report to enrich and runs at the Level 0 floor by construction, so the levers that move it are breadth, run-length policy and input complexity below. That floor also locates the field’s constraint: with nothing supplied, locating the defect is where the capability is weakest, and Frontier AI for Vulnerability Discovery develops that position across discovery, exploitation and end-to-end repair.

Reasoning budget

Reasoning budget is a per-model lever and has to be measured per model. On a random 300-task subset, about a fifth of the benchmark, the publisher compared thinking and non-thinking modes for Qwen3-235B-A22B, GPT-5, Claude-3.7-Sonnet and Claude-Sonnet-4. Most models gained modestly. GPT-5 rose from 7.7% at minimal reasoning to 22.0% at high reasoning and overtook Claude-Sonnet-4 in the process.² A pipeline that fixes one reasoning setting across a fleet forfeits a near-threefold gain on some models and spends tokens for a small gain on others.

Input complexity

Reproduction success falls as the ground-truth proof-of-concept grows, and most of the corpus sits on the failing side of that curve. Instances whose ground-truth PoC falls in the 0-to-10-byte range have the highest reproduction rate. The rate falls as that length grows and reaches roughly 10% for instances whose PoC exceeds 100 bytes, which is 65.7% of the benchmark.² The publisher attributes the fall to input-parsing complexity: a long PoC means the target parses a structured format the agent has to reconstruct before it can reach the defect.

Two consequences follow. Structured and grammar-aware input synthesis is where the remaining engineering sits, because the failing majority of the corpus demands it. And any headline reproduction rate is set in large part by how much short-PoC tail its task set holds, so two rates are comparable only when their corpora are.

Run-length policy

Most of a run’s successes land in the middle of its step budget rather than at the end. Running OpenHands with Claude-4-Sonnet under a 100-step cap, successes concentrate between steps 20 and 80 and peak between steps 20 and 50, while nearly half of all runs consume steps 80 to 100 and finish without a reproduction.² The publisher reads the 100-step cap as a balance between solving capacity and resource use. For a pipeline operator the same distribution reads as a scheduling rule: cap each attempt near the point where success density collapses, then spend the remaining budget on a fresh attempt with different retrieval, a different seed or a different model.

Breadth campaigns

The same loop finds unknown defects when it is pointed at current code instead of at benchmark instances. OpenHands was given only the latest codebases across 431 OSS-Fuzz projects. GPT-4.1 produced 16 crashes and 7 confirmed zero-day vulnerabilities. GPT-5 produced 56 crashes and 22 confirmed zero-days, 4 of which GPT-4.1 had also found.² The publisher states that CyberGym performance correlates strongly with real-world vulnerability discovery capability, and offers this campaign as the evidence for it.

Crashes converted to confirmed defects at 7 of 16 and at 22 of 56, so a breadth campaign is worth grading on confirmed defects per project scanned rather than on crash count. Crash volume rises with model capability and with scan breadth; triage capacity does not.

Measurement

The design below instantiates Evidence Centered Benchmark Design on one capability: each validation step is chosen so that passing it is evidence for the specific claim the score makes. Path attribution below is where the step and the claim separate furthest.

Differential crash validation

The oracle for a reproduction claim is a pair of runs rather than a single crash. A proof-of-concept succeeds when it triggers the target crash on the pre-patch build and does not trigger it on the post-patch build.² The patch is the discriminator: it holds the codebase fixed and varies only the defect, so a generic crash, a build failure or an unrelated memory fault fails the second run. The check is deterministic and needs no human triage, so the corpus can hold 1,507 instances and a passing score means the agent reproduced this defect.

The oracle needs a patch to discriminate against, so it grades a defect somebody has already found and already fixed. An open-ended run has no post-patch build and therefore no discriminator. Path attribution below carries the confirmation step it uses instead: re-validation against the latest release, then manual inspection of what the surviving proof-of-concept hit.

The input-level ladder

A reproduction rate carries its input level, because the four levels grade the same corpus and differ only in how much of the answer the task supplies. Level 0 reproduces 3.5% of instances, and each level above it adds context the publisher reports as greatly increasing that rate.² Running Level 0 through Level 3 converts one score into a curve, and the curve locates an agent between two very different tasks: finding a defect unaided, and reconstructing one from a description, a stack trace and a patch. Reporting a single number from an unstated level tells a reader nothing about which of those the agent can do.

Path attribution and the off-target finding

Reproduction runs also produce results outside the task they were set, and those results are worth more than the score. Among the PoCs that crashed already-patched builds, manual inspection confirmed 17 cases of incomplete patches spanning 15 projects, none of which affected the latest releases. Further validation found 35 PoCs that still crashed the latest versions; after deduplication and analysis these corresponded to 10 unique previously unknown zero-day vulnerabilities, each of which had persisted for an average of 969 days before discovery.² CyberGym Benchmark carries the full campaign figures.

An oracle that asks only whether the assigned defect reproduced discards all of it, because every one of those findings is a crash on a build the oracle expects to be clean. Recovering them took two mechanisms the benchmark score does not contain: re-validation of post-patch crashes against the latest release, and manual inspection of what the surviving PoCs actually hit. End-to-End Harness Evaluation states the general rule and carries the matching result for exploitation and for end-to-end patching.

CyberGym’s headline counter reports 34 zero-day vulnerabilities. The zero-day components it names sum to 35: 10 from the reproduction runs, plus 7 from GPT-4.1, plus 22 from GPT-5, less the 4 that overlap. The publisher does not reconcile the difference.²

Limits

Because differential validation grades defects that were already found and already fixed, benchmark performance is a proxy for field yield, and the breadth campaign above is the evidence these sources offer for the proxy holding. CyberGym Benchmark records the instrument’s own caveats, including harness-format mismatch, the OSS-Fuzz corpus bias toward C/C++ memory-safety classes, and public-corpus training contamination.

Two caveats travel with any published score. The publisher states that results are evaluated and submitted by individual teams and that agent runs are stochastic, so scores vary between evaluations. It also states that vulnerability descriptions can be ambiguous and that, with leading systems already scoring highly, modest score differences may not reflect a real capability gap.² Two leaderboard labels change what a row measures rather than the task’s difficulty: dynamic marks an agent running against a sanitized vulnerable Docker image or the compiled target binary, and test-time mem. marks an agent carrying a test-time-updated knowledge base across instances.²

Placement

The discovery leg feeds a remediation function. VulnOps: Vulnerability Operations owns that function and the triage discipline that receives this stage’s output, and VulnOps Implementation Roadmap (Enterprise) builds it out. Autonomous Exploit Generation owns the next stage, where a reproduced crash becomes controlled execution, and it carries its own method and measurement. AI Vuln-Discovery Benchmark Landscape compares the instruments that score this practice.

Notes

Footnotes

1. UC Berkeley RDI runs three benchmarks over the vulnerability lifecycle: CyberGym for reproduction, ExploitGym for exploit generation, and CyberGym-E2E for end-to-end discover-and-patch (all fetched 2026-08-31). Local copy of the index: .raw/articles/cybergym-observatory-2026-08-31.md. ↩

2. UC Berkeley RDI, CyberGym (fetched 2026-08-31). Published at ICLR 2026, OpenReview 2YvbLQEdYt; preprint arXiv:2506.02548. Local copy: .raw/articles/cybergym-benchmark-2026-08-31.md. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³