Rethinking How We Evaluate Security Agents for Real-World Use
A practitioner talk by Mudita Khurana (Airbnb) at Unprompted (March 2026) arguing that security-agent evaluation is rooted in too-narrow, outcome-only benchmarks. Abstract-only; slides and video are not yet captured.
The Argument
Outcome-only benchmarks tell whether an agent produced a correct answer, but not how it arrived there or whether the behavior will stay stable once deployed. In practice, security is a connected, end-to-end workflow: a find → confirm exploit → patch → validate loop. Agents that score well on task-specific benchmarks often fail in multi-stage settings because of contextual loss and brittle transitions between steps. The talk proposes a capability-centric framework that emphasizes observability into how agents plan, reason, use tools, and carry context across the security lifecycle, so teams can judge real-world readiness rather than single-answer accuracy.
Where It Fits
This is the practitioner counterpart to the buyer-side and benchmark-side evaluation work the wiki already tracks. It aligns with Gartner’s evaluation framework (outcomes and reasoning quality over “alerts processed”) and explains why a single-task scoreboard like DefenseBench’s BOTSv3 is necessary but not sufficient: a multi-stage loop needs multi-stage evaluation. Together these three — Gartner’s criteria, DefenseBench’s scores, and Khurana’s capability-centric, observability-first method — frame the evaluation gap in the Agentic SOC: State of the Field thesis.