Vulnerability Research in the Agentic Age
Source: Black Hat USA 2026 keynote · session listing
Yan Shoshitaishvili, associate professor at Arizona State University and author of the angr binary-analysis framework, delivered the closing keynote of Black Hat USA 2026 on the conference’s final day, the same event at which OpenAI presented its incident reconstruction.1
The argument is that the field’s comparative evidence about AI vulnerability discovery measures the wrong thing. Alongside it the talk reports a three-configuration comparison, run on one fixed model generation, that separates what the harness contributes from what the model does.
The keynote scopes out AI-hallucinated bug reports entirely. Shoshitaishvili references Apple curtailing its vulnerability research award program under a flood of low-quality AI submissions, then states the assumption the talk runs on: that filtering hallucinated findings is a solved problem in the hands of researchers who know the domain, as it was for fuzzing before it.1
Three routes to autonomous discovery
Every headline of the form “approach X found N bugs” is one of three moves, and the talk separates them because they have different implications for what was learned:
- Analyze better. Same target class, stronger analysis — fewer false positives, faster before the timeout. Mythos is the agentic-age instance.
- Analyze more. Same analysis, wider net. The lab’s Arbiter platform, built on angr, analyzed every program shipped in distribution repositories rather than a curated sample.
- Analyze differently. Change how the analysis works at all and new bugs fall out, because bugs are everywhere.
The third move is where the talk’s argument lives, and it is the one most easily mistaken for the first.
The analyzer as cookie cutter
Vulnerabilities in a program are rolled-out dough; an analyzer is a cutter that stamps out one shape and leaves the rest. Run a static analyzer, fix everything it reports, run it again: no new findings. The code is not clean. That cutter’s shape is exhausted.
A second prototype that finds a small additional set is not better than the first — it ran second. Build prototype A on angr, then build a slightly different prototype B, and B finds a marginal increment. Invert the order and the increment moves with it. Substitute a manual audit for the static analyzer and the same inversion holds: whichever pass runs first harvests the overlap, and the second reports the residue.
The talk applies this to two successive waves of the field’s own literature. The decade since the DARPA Cyber Grand Challenge produced a run of fuzzers each claiming to beat the last on already-fuzzed code, a comparison the ordering effect largely explains. LLM-based discovery is now making the identical claim against fuzzers: an LLM run over well-fuzzed code finds many bugs, the fuzzer re-run on the same code finds none, and the field concludes the LLM is the stronger analyzer. Fuzzing is the partial exception, and it locates the mechanism: because fuzzing is stochastic, re-running the same fuzzer does keep finding corner cases, which deterministic static analysis does not.
The corrective experiment requires rewinding the timeline and analyzing from scratch. The lab is attempting it and reports it is very difficult, for a reason specific to this generation of tooling: training contamination. The model already knows the bugs the fuzzer found. This is developed on Analyzer Ordering Confound.
Vulnerability properties
The shape a cutter stamps out is made of properties: implicit attributes of a vulnerability such as arising from a data flow, requiring injection of attacker-controlled content, or depending on multi-threaded behavior. Static analyses target properties explicitly. Fuzzers and unstructured LLM prompting target them implicitly, which is why their coverage is hard to characterize.
The keynote’s engineering proposal is to extract properties from well-studied software and apply them to software that has not been studied. Three results carry it, covered in full on Vulnerability Properties.
OpenHarmony. Huawei’s OpenHarmony is a from-scratch reimplementation of an Android-inspired operating system running on a billion devices, and the talk states no major academic security study of it existed before the lab’s paper, due out the week after the keynote. The method mined a decade-plus of Android security research into a recipe book of vulnerability properties, with the mining step agentically assisted, and then used those properties to guide manual review of millions of lines of OpenHarmony code. Roughly every property applied specifically produced a new zero-day: dozens of flaws spanning Bluetooth device takeover and location and privacy leaks. An agentic version of the pipeline is running, with results held for a later talk.1
The Linux kernel scaling experiment. Arizona State University misconfigured an OpenAI agreement and granted every employee unmetered Codex access, which the lab drew on until the university noticed roughly two million dollars of spend and revoked it. Against a benchmark of 479 Linux-kernel vulnerabilities attributed to Mythos in a June 2026 Washington Post report, the lab ran three configurations on a fixed, last-generation model:
| Configuration | Triaged findings |
|---|---|
| Dozens of last-generation GPT instances, no workflow | ~300 |
| Workflow added (adversarial review, planning), narrowed to three instances | ~600 |
| Vulnerability properties added, formatted for the pipeline | over 1,000 |
Every ASU count is a local privilege escalation candidate triggerable by an unprivileged user, and all were triaged as real. The talk states the comparison against the 479 figure is apples-to-oranges, since published counts typically include root-only bugs and the Mythos composition is not documented.1
Rust rewrites. Properties survive reimplementation. Using the same unmetered access, Shoshitaishvili had agents reimplement load-bearing C libraries — libssl, libpng, libxml among them — as millions of lines of Rust behind an assurance pipeline. Memory-corruption classes did not survive the language change. The rest did: the crypto library reproduced the classic non-memory-safety cryptographic attacks the original was vulnerable to, including under explicit instruction naming those historical vulnerabilities and forbidding them. His framing is that certain kinds of code are predisposed to certain kinds of vulnerability independent of the language expressing them.
The real-world confirmation is not his own project. Seventy-nine CVEs dropped against a Rust reimplementation of coreutils after it shipped in the current Ubuntu release — no memory corruption, and time-of-check/time-of-use flaws in critical utilities instead. The lab’s property-aware engine finds more in the same codebase. Agent-driven formal verification to eliminate logic errors is named as a research direction, explicitly not a present capability at scale.1
Disclosure at ten times the reportable rate
The lab finds vulnerabilities roughly ten times faster than it can report them, because a report that is worth sending carries a proposed fix and an analysis rather than a bare finding. The constraint is human effort on the triage-to-report path, which is the same bottleneck VulnOps names and the Glasswing one-month update reported from the discovery side.
The keynote’s contribution is that the disclosure process was already net-harmful for one device class before agentic scale arrived. A May 2026 paper from the lab, on embedded devices and without agents in the loop, reproduced disclosed vulnerabilities against other devices held in the lab. The finding: disclosing one vulnerability endangers roughly three times as many devices as it secures, with the exact ratio depending on what is counted as endangerment.1
The stopgap in flight is a public site carrying over a thousand local-privilege-escalation bugs, publishing CVE details for the disclosed ones and hashes for the remainder, with vendors and security groups engaged on what proactive defense should look like. Candidate agent-generated patches applied ahead of vendor fixes is one option under discussion. The keynote states without qualification that the lab does not have a better solution and is taking small steps toward finding one.
No replacement disclosure model is proposed
The talk establishes that coordinated disclosure fails at agentic discovery rates and that it was already negative-sum for embedded devices at human rates. It proposes no replacement — the hash-publication site is named as a stopgap, and full disclosure is raised and dismissed. The wiki carries no source proposing a disclosure model calibrated for four-digit annual discovery from a single lab.
Model restriction and human skill
Two positions close the talk, both stated as the speaker’s opinion rather than as findings.
Restricting model capability is misguided. The argument is from fifteen years of offensive research conducted in the open: angr is open source, as is the cyber reasoning system the team built for the Cyber Grand Challenge and the AI Cyber Challenge. Asked repeatedly whether that is responsible, his answer is that the defensive population is the larger one and needs the benefit. A capability gate diminishes the positive impact more than it diminishes the negative. He extends this to restricting capable open-weight models and next-generation models.
Agentic tooling raises the return on human expertise. The step from unstructured prompting to a property-aware pipeline, worth more than 3× in the table above, came from human judgement about threat models and the vulnerability space rather than from model capability. His formulation: until agents do 100% of the work, taking 90% of it lets a researcher have more impact rather than less. He runs pwn.college, an open security-education platform with about 10,000 people actively learning on it monthly, and reports that whether security remains worth learning is the question he is asked most often.
The autonomous sharpening pipeline
The closing thesis concerns academic research prototypes rather than production tooling, and it is the part of the talk with the longest reach.
Academic prototypes have historically been evaluated on sample sizes small enough to admit unexamined assumptions, and those assumptions are why prototypes so often fail outside the paper. His own first firmware paper, eleven years earlier, evaluated authentication-bypass detection on three samples. A successor from the same lab scaled to a thousand firmware images — fewer implicit assumptions, though the talk is explicit that many remain.
What agents change is not the analysis but the evaluation loop around it. An agent that can drive a computer can run a prototype against new samples, ground each failure in an observed behavior rather than a guess, and feed that back as a fix to the prototype itself. The result is a self-improving system that scales the dataset and the results together, which the talk names the future of vulnerability research, expects to be transformative over the next two years, and states will take sustained human effort to build.
Bounded by METR's 2025 RCT
This keynote reports large gains from agentic scale-out and predicts more. METR’s randomized trial measured experienced open-source maintainers working on their own repositories as roughly 19% slower with early-2025 AI tools. The two are compatible but only under a reading worth stating: the keynote’s gains come from throughput on work no human was doing at all — a thousand firmware images, every binary in a repository — while METR measured expert humans on their own familiar code. Neither source tests the other’s regime, and no source on the wiki measures a property-aware pipeline against an expert human on the same corpus.
Sourcing caveats
The keynote is a primary account of unpublished work, and the wiki’s citation of it inherits that status.
- The OpenHarmony paper was not published at talk time. Its figures — dozens of zero-days, a billion devices, roughly one zero-day per property applied — rest on the speaker’s statement alone.
- The Linux-kernel counts are the lab’s own triage, with no external verification and no published corpus.
- The 479-vulnerability Mythos figure is a press report, not a vendor publication, and the talk itself flags that exact Mythos numbers are hard to obtain.
- The transcript is machine-generated. Student names are transcribed phonetically and are not reproduced here as attributions.