CMM D7: Observability and Detection

Agentic AI Security CMM — D7 Observability & Detection (Deep Dive)

Companion deep-dive to the CMM’s D7 domain, written under the recalibration method. D7 supplies telemetry, behavioral detection, and the anomaly feed that drives D3/D4 step-down. The detective half of OWASP Agentic AI Threats and Mitigations lands here: D7 is where threats that only become visible through monitoring are caught — Memory Poisoning (T1), Repudiation and Untraceability (T8, addressed by immutable per-agent logging), and the multi-agent threats Agent Communication Poisoning (T12), Rogue Agents (T13), Human Attacks on Multi-Agent Systems (T14), and Human Manipulation (T15), whose Playbook 6 (multi-agent communication and trust) specifies message authentication and cross-agent anomaly detection. Two facts shape the whole domain. The standard telemetry layer (OpenTelemetry GenAI) is not yet stable. And D7 is the run-rate-heavy domain: agent logs run roughly 10–20× human volume into the SIEM, so cost scales with agent count, not control coverage.

D7 is the CMM domain that NIST AI 800-4 maps most directly: the report is a descriptive landscape of the gaps, barriers, and open questions in monitoring deployed AI systems, and it is the requirements- and gap-source for this domain rather than a control standard. Its cross-cutting challenges — immature trusted methods, the lack of direct visibility into model properties, and the agent-specific items (agent-identifier standardization, monitoring multi-agent distributed systems, detecting deceptive or monitor-evading behavior) — name the open problems the upper levels here run into, and AI 800-4 supplies no mappable controls to close them.

Single-source grounding

Levels and cost model synthesize the recalibration method against the regulated-FI stress test plus vendor documentation. Tooling status is a May 2026 snapshot.

Threat coverage

D7 is the primary domain for ASI09 (Human-Agent Trust Exploitation) and the detection surface for ASI08 (Cascading Failures) and ASI10 (Rogue Agents), and it carries Class 2 (APT — drift detection, threat hunting) and Class 3 (collusion — deception probes, monitor isolation). Per-agent baselining is capped by D2, and the collusion and cascade detectors remain research-stage. See the Threat Taxonomy Reconciliation matrix and the threat classes.

Control landscape (dated)

Capability	What ships today	Status (May 2026)	Platform-native (MS / AWS / GCP)
GenAI telemetry semantic conventions	OpenTelemetry gen_ai.* SemConv	Development / experimental — not yet stable1	All three emit/consume gen_ai.* (Azure Monitor, AWS ADOT, GCP Cloud Trace)
Glass-box telemetry (hooks + spans)	coding-tool hooks; OTel agent/tool spans	hooks GA; agent spans experimental but stable in practice	MS: App Insights Agents view2. AWS: AgentCore emits OTel. GCP: ADK emits OTel
Per-agent identity multiplexing of logs	inject agent_id / session / trace_id (pattern)	GA where identity exists	depends on D2 — this is the D2→D7 cap
Behavioral monitoring / drift detection	Miggo DeepTracing, Vectra (COTS); UEBA-for-agents	COTS GA	MS: Defender XDR AI-agent detection — preview, Agent 365-licensed3. AWS/GCP: no native behavioral-drift detector
Anomaly detection (runaway / recursive / resource)	Datadog, New Relic, Sentry (COTS)	COTS GA	MS: Defender RTP (preview). AWS: CloudWatch GenAI Observability — preview4. GCP: Cloud Trace dashboards
SIEM integration + agent-aware playbooks	Splunk; agentic-SOC playbooks	varies	MS: Sentinel data lake + MCP server GA; graph MCP tools + entity analyzer sign-up preview5
Forward-pass / activation monitoring	mechanistic-interp activation monitoring	research-grade, pre-launch	none

The recalibration corrects two things in the current D7 tooling presentation. It adds the OTel-not-stable caveat at L3. And it re-grades the Microsoft-native behavioral detector (Defender XDR AI-agent detection) as preview and Agent 365-licensed, mirroring the D4 “L4 looks GA but isn’t” correction. Google SecOps adds a second platform-native behavioral detector, Agent Anomaly Detection, which pairs statistical models with an LLM-as-a-judge check on agent reasoning (preview; see Google Agentic SOC).

The Microsoft ZT4AI Visibility / Orchestration pillar (assume breach) supplies the Microsoft-native detection and SOC controls behind these rungs — Defender XDR AI-agent detection and the Sentinel agentic-SOC tooling — crosswalked to D7 in the 2026-Q2 ZT4AI review, which records the same preview status. Above those controls, Agent 365’s observe layer — the agent registry, Registry sync (including cross-cloud agents), and the Agent Map — is its strongest native D7 contribution: an agent-inventory and telemetry surface that aggregates the underlying detections rather than replacing them. The inventory-versus-detection distinction, and the absence-claim that the observe layer inventories but does not verify supply-chain integrity (no AI-BOM), are set out in Agent 365 review.

Capability-decoupled levels

Stated as capabilities per rule 1; a control counts when it operates in production per rule 2.

• L1 — Initial. No agent-specific telemetry; only the vendor console.
• L2 — Developing. A tool-call audit log records action history with user attribution.
• L3 — Defined. Agents emit OpenTelemetry gen_ai.* spans (model / request / tool / retrieval / agent); a trace backend sits in-path; logs carry per-agent identity multiplexing (every action traces to an agent identity and the invoking human); every tool call meets a minimum action-log schema with a recoverable rollback reference. Maturity note: the underlying SemConv is experimental, so pin the convention version to keep a bump from silently breaking detection rules.
• L4 — Managed. Per-agent behavioral baselines / drift detection operate in production and wire to the SIEM/SOAR; AI-SPM posture monitoring is deployed; a multi-category eval cadence runs (single-tool coverage is not L4). Maturity caveat: the Microsoft-native behavioral detector (Defender XDR AI-agent detection) is preview and Agent 365-licensed, so an L4 program today assembles COTS, preview-native, and OSS pieces.
• L5 — Optimizing. Agent-aware SIEM playbooks run in production; per-agent baselines hold a documented prompt-volume-to-alert ratio for at least a quarter; the analyst-actionable alert rate is measured and high; every triggered alert wires to a controls update within an SLA, closing the loop into D3/D4 step-down.
• L5+ — Leading Edge. A cascade-detection rule library with tuned thresholds for multi-agent (ASI07/08/10) risk (research-stage, no shipping library); cross-agent joint-distribution baselines; model forward-pass activation monitoring (no shipping product).

The level ladder is already capability-shaped and grades L5+ as research-stage; the recalibration strips embedded product names to the tooling map and adds the maturity caveats at L3 and L4.

Right-sizing by deployment shape

Deployment shape	Realistic D7 target	Why
Internal RAG / support chatbot (no/few tools)	L3	OTel spans + identity multiplexing + action log; near-zero tool surface needs no behavioral-drift or multi-tool red team. The persona’s bot sits here
Data-science / coding copilot	L3 → L4	Long-running sessions and tool writes justify per-agent drift baselines and an eval cadence
MCP / skill provider serving others	L4	Third-party blast radius; MCP-protocol-aware anomaly detection and AI-SPM become first-order
High-autonomy multi-agent mesh	L4 → selective L5+	Cascade / rogue-agent detection and joint-distribution baselines become load-bearing only here

The lethal-trifecta test is D7’s strongest right-sizing lever. A program with strong D3/D4/D5 architectural containment may legitimately run no behavioral-observability layer and still be sound (the Stripe containment pattern). A contained design scoring low on D7 records the choice as an intentional trade-off.

Cost model

Level	Licensing	Operational labor	Run-rate (dominant)
L2	~0 (E5: Sentinel/Defender entitled)	~0.25 FTE: turn on tool-call audit	low
L3	~0 for an E5 incumbent (App Insights / Azure Monitor + Sentinel ingest entitled; OTel SDK is OSS)	~0.5 FTE: span instrumentation, identity-multiplex wiring, schema conformance	the inflection — gen_ai.* trace volume hits the SIEM at roughly 10–20× human log volume
L4	mostly ~0 native, but Defender AI-agent detection needs an Agent 365 license; Miggo/Vectra are off-stack COTS otherwise	baseline tuning, drift false-positive triage, multi-tool eval ops	behavioral-analysis compute + full agent-log retention, scaling with agent count
L5	some off-stack (real-time AI-BOM, CART tooling)	continuous baseline-ratio maintenance, actionable-rate measurement, SLA runbook	peak: every-surface telemetry × every agent into the SIEM

The dominant cost signal: licensing is near-zero through L3 for an E5 incumbent, and the spend is ingestion run-rate that scales with agent count. The recalibration adds the mitigation the current D7 omits, log tiering: route high-volume, low-fidelity agent trace spans to a cheaper data-lake or auxiliary tier, and reserve the expensive analytics tier for detections that fire.⁶ The one licensing cliff is the L4 native behavioral detector, which requires Agent 365 rather than classic E5.

Customer critiques folded in

• “Switch on logging we already pay for.” Confirmed: D7-L3 is near-zero licensing for the persona; the work is instrumentation labor.
• “Log-ingestion run-rate is the dominant cost.” Confirmed and quantified, with the tiering mitigation named, a lever the current D7 does not surface.
• “L5 names just-GA’d products.” Defender AI-agent detection (preview) and Sentinel graph MCP tools (sign-up preview) should not anchor a GA-grade L4/L5 for a regulated buyer until production-hardened.
• “No behavioral baselines yet; PyRIT covers one red-team leg.” Fair: the persona’s bot targets L3, and L4 requires multi-category coverage. A single tool is not L4.

Open questions

• OpenTelemetry GenAI conventions remain in development; a version bump can break detection rules, and there is no confirmed stable-release date.
• Defender AI-agent detection is preview and Agent 365-licensed; whether classic-E5 buyers get it at GA without Agent 365 is the variable that moves the native L4 from preview-assembled to turnkey.
• AWS and GCP emit OTel telemetry natively but ship no native behavioral-drift detector; single-stack buyers there go off-platform for L4.
• The published analyst-actionable-rate benchmark comes from a single hyperscaler-scale SOC; whether it generalizes is unverified.

D2→D7 dependency cap

D7’s effective score is capped at D2’s raw score (effective(D7) ≤ raw(D2)): without a verifiable per-agent identity, per-agent behavioral baselining has no stable principal to bind to, and logs collapse to a shared principal. For the persona (D2 at L2), D7’s effective score is capped at L2 regardless of observability spend. The scoring guidance follows: the cheapest high-leverage move is D2-L3 (Entra Agent ID, near-zero licensing), which lifts the D7 ceiling. Buying a monitoring product against an unbindable identity does not. Report D7 as raw + effective. See the dependency rules.

Notes

Footnotes

1. OpenTelemetry — GenAI semantic conventions, 2026. Status: Development; not yet stable. ↩

2. Microsoft Learn — Monitor AI agents with Application Insights, 2026. OTel GenAI-based agent observability (preview blade). ↩

3. Microsoft Learn — Detect and investigate threats to AI agents with Defender, 2026. AI-agent detection, preview; Agent 365 licensing. ↩

4. AWS — Amazon CloudWatch generative AI observability (preview), 2026. ↩

5. Microsoft — Sentinel MCP server generally available, 2026. Sentinel graph GA; graph MCP tools / entity analyzer in sign-up preview. ↩

6. Microsoft Learn — Plan costs and understand Microsoft Sentinel pricing, 2026. Analytics vs auxiliary vs data-lake ingestion tiers; entity-analyzer Security Compute Units. ↩