Adversarial Reflexion

Adversarial Reflexion (Constrained-Persona Vulnerability Verification)

Adversarial Reflexion is an agentic vulnerability-verification technique that wraps the LLM in a tightly constrained attacker persona and forces it to trace each exploit step explicitly against the real codebase via tool use. The term is borrowed from the Shinn et al. 2023 Reflexion line of work on LLM self-critique with explicit memory traces; the adversarial specialization is named and operationalized in Knostic’s OpenAnt announcement (2026-05-15) for verified vulnerability discovery in open-source codebases.

The Failure Mode It Addresses

The naive verification pattern — “You are an attacker. Can you exploit this?” — is structurally unreliable because LLMs are agreeable by default. Asked “is this code vulnerable?” the model will find a way to say yes; asked “can you exploit this?” the model will construct a plausible-sounding scenario that assumes capabilities the attacker does not have (server access, admin credentials, ability to modify files, local shell access). This is the LLM-as-a-Judge sycophancy failure mode applied to attacker-role exploit confirmation, and it produces a steady stream of technically-correct-but-practically-meaningless findings that drown the signal.

The Mechanism

Three components, all enforced as architectural constraints at the harness layer rather than as prompt-engineering best-effort:

1. Capability removal. The model is told it has no server access, no credentials, no local-file access. For CLI tools and libraries, the constraint is sharper: no ability to run CLI commands — the exploit must be triggered remotely. If the only attack path requires local execution by a privileged user, the finding is classified not exploitable (the rationale: local users can already do anything on their own machine; a “vulnerability” that only fires under local-user privileges is not a vulnerability under realistic threat-model assumptions).
2. Explicit step-by-step tracing. The model must show the specific input, the specific endpoint, the specific data flow. Hand-waving over hard steps is structurally disallowed; the harness rejects findings whose exploit trace skips steps.
3. Tool-use verification against the real codebase. The model is given tool access to search the codebase, read related functions, and trace exploit paths. Claims that contradict the actual code fail verification at the agentic level rather than at human-review time.

The composite effect is to eliminate a structural class of false positives — the “the model played along with my framing” class — without requiring the model itself to become less agreeable. The model can still be sycophantic; the harness will not honor sycophancy as evidence.

Generalization beyond OpenAnt

Adversarial Reflexion is OpenAnt’s specific implementation, but the underlying discipline — FP-control via architectural constraint at the harness layer — is convergent across AI vulnerability discovery generally and across at least two surfaces (vuln discovery + config audit):

• MDASH runs the discipline as ensemble + debater + prover-stage architecture: multiple independent perspectives evaluate the same candidate, and confirmation requires consensus.
• For Codex Security the mechanism is sandboxed exploit-trigger validation — each candidate vulnerability is attempted in an isolated sandboxed environment to confirm exploitability, with the validation steps described to support quality assessment.
• Claude Code Security implements self-critique prove/disprove verification: “Claude re-examines each result, attempting to prove or disprove its own findings and filter out false positives.” The model-vs-itself adversarial loop is the FP-control primitive.
• XBOW × Mythos draws the same line through live-site validation, the wedge between finding a candidate and confirming a live-site exploit.
• On the config-audit side (a different domain), AgentShield reaches the analogous discipline via provenance-aware runtimeConfidence weighting — same rule, different weight by source kind (active-runtime vs. template-example vs. docs-example).

Instrument	Vendor	Domain	Mechanism
AgentShield	Affaan M / ECC	Agent config audit	Provenance-aware finding-weight by source kind
OpenAnt	Knostic	App-code vuln discovery	Constrained-attacker-persona + explicit trace + tool-use verification
Codex Security / Aardvark	OpenAI	App-code vuln discovery	Sandboxed exploit-trigger validation
Claude Code Security	Anthropic	App-code vuln discovery	Self-critique prove/disprove
MDASH	Microsoft	App-code vuln discovery	Ensemble + debater + prover-stage
XBOW × Mythos	XBOW	Live-web exploit	Live-site validation
security-audit-skill	Cloudflare	App-code vuln discovery	Phase 3: a second independent agent tries to falsify each finding
VVAH	Visa	App-code vuln discovery	Stage S6: a second independent agent tries to falsify each finding
skills	Trail of Bits	Multi-domain code review	fp-check: a second independent agent tries to falsify each finding
defending-code-harness	Anthropic / Semgrep	C/C++ memory safety	Fresh-container grader: a second independent agent tries to falsify each finding

Six of the ten instruments above are vendor products; the other four are open-source releases from four separately owned projects — Cloudflare’s security-audit-skill, Visa’s VVAH, Trail of Bits’ skills marketplace, and Anthropic’s defending-code-harness, now forked and maintained by Semgrep — and Semgrep’s July 2026 survey of the open-source field reports the technique as widely adopted for increasing accuracy.¹ The instruments also divide on which model performs the disproof. Semgrep reports the technique working especially well when a different model attempts the disproof, and finds the split falling along category lines: almost every SAST+LLM hybrid separates discovery from validation and many run separate models across the two stages, while harnesses driving to a crashing end-state tend to use one model for both.¹ VVAH is the separated case, running Sonnet 4.6 for detection and Opus 4.8 for remediation and validation, per Semgrep’s LLM-generated summary. Self-critique, where one model prove-disproves its own finding, is the same discipline with the weaker independence guarantee, because the reviewer inherits the reasoning it is checking.

Across ten sourced instruments, the mechanism varies but the disciplinary observation is identical: the agreeable-judge failure mode arises from the structure of agentic verification stages rather than from prompt phrasing, and the production-grade response removes the cheap-yes path at the architecture level rather than coaxing the model into saying no. Convergence across two structurally different groups — six commercial products and four open-source releases under separate licences — carries this further than the vendor count alone did. As of 2026-05-15 the discipline is sourced widely enough that it should be treated as established — a maturity expectation, independent of any one vendor’s design choice. See the parent CMM page’s Revision-pass candidates info callout for the §What is now established split.

Naming Note

The concept is sometimes called constrained-persona verification or capability-constrained adversarial role-play. The “Reflexion” framing borrows from Shinn et al. (the original Reflexion paper uses explicit-memory self-critique for general-purpose LLM-task improvement); OpenAnt’s specialization is the adversarial application — the LLM is critiquing its own exploit attempt under hard capability constraints rather than critiquing its own task solution. Knostic’s announcement is the source for Adversarial Reflexion as a named technique; the wiki adopts that name while noting the mechanism is the load-bearing part.

Relationship to Existing Wiki Coverage

• LLM-as-a-Judge — the failure mode this technique addresses. The wiki’s existing concept page treats LLM-as-a-Judge as a generic agreeable-model evaluation pattern; this concept names the specific attacker-role sub-class and its harness-level mitigation.
• Control-Efficacy Gate — adjacent FP-control instrument from the AgentShield ingest; corpus-gate (positive-test regression) and exception-lifecycle (negative-test waiver audit) are both control-efficacy checks operating at CI time, while Adversarial Reflexion is a verification-architecture technique operating at inference time. Both reduce false-positive surfaces by structural means; they are not the same primitive.
• Harness Config as Supply-Chain Artifact — sibling generalization from the AgentShield ingest; covers a different domain (config audit, not vuln discovery) but operates under the same FP-control-as-architectural-primary discipline.
• Frontier AI for Vulnerability Discovery — the thesis Adversarial Reflexion adds a fourth sourced production path to. See thesis §How-This-Has-Evolved for the 2026-05-15 entry.
• CMM Measurement Protocol — cites this technique’s constrained-persona architecture as the worked example of the “production-grade” answer to its Stage 2 interview question on how a detector’s false-positive class is controlled.

See Also

• OpenAnt · OpenAnt announcement — source.

• Shinn et al., Reflexion: Language Agents with Verbal Reinforcement Learning (NeurIPS 2023) — the prior-art Reflexion technique OpenAnt borrows from. Not yet on the wiki as a paper page.

• RAPTOR — structurally the offensive twin to OpenAnt’s use of this mechanism: where OpenAnt applies constrained-attacker-persona FP control inside a discovery pipeline, RAPTOR carries the same pipeline further into exploitation.

Notes

Footnotes

1. Semgrep — Comparing open source AI code security harnesses, July 2026 (no day-level date exposed; author not named). The adversarial-validation instance names and the different-model observation are human-written; the VVAH model split is from Semgrep’s LLM-generated repository summary. Summarized at OSS AI Security Harness Comparison. ↩ ↩²