Enterprise Security in the Agentic AI Era

Red Teaming for AI: Synthesis

3 min read

Red Teaming for AI: Synthesis

On this page

Question

What does a complete red-team practice for AI applications look like in 2026, across probe libraries, orchestration, regression suites, and continuous adversarial testing? Specifically: which tools cover which quadrants of the four-quadrant red-team grid from CMM D7 L4? What are the trust and provenance assumptions behind each? How does evaluation methodology (CLASP, ECBD, LLM-as-a-judge) tie back into the practice?

Current position

This is the most mature of the wiki’s new scope axes; the substrate was built before the scope expansion. The four-quadrant red-team coverage codified in CMM D7 L4 is the working decomposition:

  1. Probe libraries. garak (NVIDIA) is an OSS LLM vulnerability scanner with 18+ probe categories spanning encoding, prompt-injection, GCG, DAN, malware generation, XSS, and leak-replay. Cross-check vendor-published numbers against garak outputs.
  2. Orchestration. PyRIT (Microsoft AI Red Team) provides multi-turn adversarial orchestration with adapters across OpenAI, Anthropic, Google, HuggingFace, and self-hosted endpoints; it is widely used as the OSS standard for orchestrated red-team campaigns.
  3. Regression suites. Promptfoo is the regression-test surface for application-layer LLM behavior, most useful as the CI gate for prompts and tool definitions.
  4. Continuous adversarial testing. Mindgard CART is a SaaS for continuous red-team across deployed models; General Analysis is an agentic-AI-specific entrant.

Evaluation methodology is the harder problem. CLASP supplies a capability-centric evaluation rubric (Planning, Tool Use, Memory, Reasoning, Reflection, Perception); ECBD provides the design methodology for benchmark construction; LLM-as-a-judge is the semantic-matching approach most evaluation toolchains converge on, with known failure modes (overconfidence, bias, prompt sensitivity).

The productized red-team-for-AI surface is consolidating around three incumbents: Lakera Guard for content-layer guardrails plus testing, HiddenLayer for AIDR with model scanning and adversarial robustness assessment, and Protect AI for AI-BOM, ModelScan, and the huntr bounty surface.

Supporting evidence

  • AgentDojo (NeurIPS 2024) is an independent benchmark for tool-using agents: 97 tasks, 629 security cases. Independent academic benchmarks are scarce across the surveyed tooling, which is what gives this one weight.
  • OWASP LLM Top 10 and OWASP Agentic AI Top 10 supply the vulnerability taxonomy that probe libraries map against.
  • OWASP AIVSS establishes a scoring framework for AI vulnerabilities, analogous to CVSS for traditional vulnerabilities.

Open questions & caveats

Model-extraction and inversion attacks in productized tooling

Model-layer attacks (extraction, inversion, membership inference) are documented as concepts but thinly represented in the productized testing surface. Across the scanners surveyed here, coverage centers on prompt-injection and jailbreak; the model-layer attack class is harder to test for and consequently under-covered.

Independent reproducibility of vendor red-team claims

Vendor-published numbers dominate; few independent reproductions exist. The AgentDojo benchmark is one of the few neutral data points among the sources surveyed.

How this position has evolved

  • 2026-05-13. Seeded as a synthesis of material previously spread across concepts/, entities/products/, and the CMM domain definitions. As an existing-content synthesis rather than an ingest-driven seed, the page is positioned to move to developing once cross-links are added to its constituent pages.

Open sub-questions

  • Is redteam-for-ai a separate scope axis, or a sub-axis of sec-of-ai that should be collapsed? Current judgment: keep separate, because the tooling and methodology surface is large enough to warrant its own synthesis address.
  • How does red-team-for-AI methodology need to evolve for agentic AI (multi-turn, multi-tool, multi-agent) versus classical LLM testing? Some of CLASP’s extensions hint at this, but the field is unsettled.
  • See Gaps Index for related open questions.

Graph View

Backlinks