CTI-REALM Benchmark
CTI-REALM is Microsoft Research’s open-source benchmark for end-to-end detection-rule generation — the defender-side counterpart to the offense-leaning exploit benchmarks. Published 2026-03-20 (arXiv 2603.13517), it measures whether an AI agent can convert cyber threat intelligence into validated detections, not merely answer a quiz. (Confidence: high on structure; medium on exact per-model scores — the blog reports ranges, not a full table.)
What It Measures
The benchmark replicates a detection engineer’s full workflow: read a CTI report, explore telemetry, write and iteratively refine KQL queries, and produce validated detection logic (Sigma rules + KQL). It scores intermediate decision quality at checkpoints — CTI comprehension, MITRE ATT&CK technique mapping, data-source identification, query refinement — rather than only the final rule. Emulated attacks span Linux, cloud, and Azure Kubernetes Service, with ground-truth labels.
Two tiers exist — CTI-REALM-25 and CTI-REALM-50 (25 and 50 tasks, atomic attacks through multi-step intrusions); the published results use CTI-REALM-50, built from 37 curated public CTI reports. The metric is a reward score on a 0–1 scale.
Results
Across 16 frontier models, Microsoft reports that Claude occupies the top three positions, with reward 0.624–0.685; Claude Opus 4.6 (High) leads at roughly 0.637, with GPT-5 variants following. The blog gives ranges rather than a full per-model table.
Per-model table not yet sourced
The blog publishes only the top range (0.624–0.685). The arXiv paper and the UK AISI inspect_evals harness — which integrates CTI-REALM — are the candidates for the full leaderboard.
Why It Matters
CTI-REALM fills a quadrant the exploit benchmarks miss: defensive detection engineering. Where CyberGym measures vulnerability reproduction and ExploitGym measure exploit construction (both offense-leaning), CTI-REALM measures whether agents can convert intelligence into deployed defenses — the SOC-automation capability behind MDASH and the agentic-SOC thesis. Microsoft open-sourced it so the industry can test models against a shared detection-engineering standard. See the benchmark landscape for where it fits. Sun’s Palo Alto pipeline is a production instance of converting unstructured OSINT threat reports into a queryable, source-cited knowledge graph.