DefenseBench

Sources: DefenseBench · BOTSv3 benchmark

DefenseBench is a public benchmark platform that evaluates how well AI agents perform defensive (blue-team) cybersecurity tasks: alert triage and incident investigation in production-like environments. As of mid-2026 it is a research preview with one active benchmark.

BOTSv3

The active benchmark, BOTSv3, is built on Splunk’s Boss of the SOC v3 dataset. Agents investigate security incidents by running Splunk searches and answering forensic questions under time constraints. The leaderboard scores each run on average and best score and on “best correct” — the number of incidents correctly solved. As of a May 2026 snapshot it ranked general coding-agent configurations run in interactive mode (Claude Opus 4.6 and GPT-5.x Codex variants near the top, smaller models lower); it therefore measures off-the-shelf coding agents on a SOC-investigation task rather than purpose-built defender agents. The leaderboard is live, so any specific ranking is a snapshot.

Why It Matters

DefenseBench is the defender-side counterpart to offensive agent benchmarks such as AgentDojo and the suites catalogued in the AI vuln-discovery benchmark landscape. Its existence narrows a gap the Agentic SOC thesis had recorded as open: that no public benchmark existed for defender agents. The gap is narrowed, not closed. BOTSv3 is a single Splunk-derived dataset (forensic question-answering under time limits); the platform is a research preview with thin published methodology; and the leaderboard currently measures general coding agents rather than the purpose-built AI-SOC agents that Gartner’s evaluation framework addresses. Even so, it is the first public, reproducible scoreboard for agentic SOC investigation, a comparator the field previously lacked.

See Also

• Agentic SOC: State of the Field — the thesis whose benchmark gap this narrows
• AgentDojo — the analogous benchmark on the attacker / agent-security side
• ADR-Bench — a different defender-side benchmark: detecting attacks against MCP agents (302 tasks, 133 servers) rather than scoring agents on SOC investigation; together they cover the two defender-benchmark axes (detect attacks-on-agents vs. agents-do-defense-work)
• Evaluating AI SOC Agents — Gartner’s buyer-side criteria (complementary: criteria versus scores)
• AI Vuln-Discovery Benchmark Landscape — the offensive-side benchmark catalogue