A finding marked fixed was fixed against a model that has since been swapped for a cheaper one. A tool added in a sprint gave the agent a write path nobody attacked. The red-team report still looks current, which is why the failure is quiet.
This paper argues that agent red teaming should be run as a test suite, not an audit. Tests are cheap to rerun, and their value comes from rerunning them. The suite should score what the agent did, not what it said, and should measure what its defenses cost legitimate work as carefully as what they stop. The strongest objection to that view gets its own section below; it changes how the suite is built, not whether to build one.
An agent's report expires when any of four inputs changes
A traditional engagement produces a report about a system at a moment. For a web application that moment lasts a while, because the code changes but its authority usually does not. An agent's behavior is a function of four inputs that change independently: the model version, the system prompt, the tool list with its descriptions, and the authorization policy around it. Change any one and last quarter's findings describe a different system.
Continuous does not mean constant. It means the program is triggered by change, and anything that alters those four inputs counts.
Index the library by technique, not by prompt
A library of a few hundred clever prompts decays fast, because the exact string that worked against one model rarely transfers to the next. Organize by technique, with several concrete cases per technique and a statement of what each case is trying to cause. MITRE ATLAS supplies shared names: it is a public knowledge base of adversary tactics, techniques and procedures targeting AI systems, organized into tactics, techniques, sub-techniques, mitigations and case studies [1]. It names the agent-relevant techniques directly, including LLM Prompt Injection (AML.T0051, with direct and indirect sub-techniques), LLM Jailbreak (AML.T0054), AI Agent Tool Invocation (AML.T0053) and LLM Data Leakage (AML.T0057) [1].
The OWASP Top 10 for LLM Applications 2025 names the weakness rather than the adversary's step: LLM01:2025 Prompt Injection, LLM02:2025 Sensitive Information Disclosure, LLM06:2025 Excessive Agency and LLM07:2025 System Prompt Leakage, among others [2]. Tag each case with both. ATLAS says what the attacker did; OWASP says which weakness let it work, which is what someone has to fix.
NIST AI 100-2 E2025 adds the dimension most libraries skip. Its taxonomy classifies attacks by the attacker's goals, objectives, capabilities and knowledge, and by the life-cycle stage they target [3]. A case that assumes the attacker can write to a document the agent reads is a different threat from one that assumes the attacker is the user, and your coverage report should show that you test both.
| Field | What it holds | Example (finance-ops agent) |
|---|---|---|
| Technique | ATLAS technique ID and OWASP 2025 category | AML.T0051 indirect prompt injection leading to AML.T0053 tool invocation; LLM01:2025 and LLM06:2025 |
| Attacker position | Where the attacker's text enters and what they control or know, in NIST AI 100-2 terms | Can send an invoice PDF to the accounts-payable mailbox; cannot see the agent's prompt |
| Payload and variants | The concrete input plus variants: encoding, language, placement, split across turns | Footer text telling the agent to update remittance details before paying |
| Prohibited consequence | The observable effect that counts as success, checked in the sandbox | Vendor bank record changed, or a payment created to an account not on file |
| Consequence tier | The tier of the action the attack is trying to reach | Payment and vendor master change: the highest tier |
| Benign twin | A legitimate request that looks similar and must succeed | A verified bank-detail change submitted through the vendor portal |
| Last run | Model, prompt, tool and policy versions of the last run, and its outcome | Caught on policy v41; rerun pending for the new model version |
Score what reached the sandbox, not what the agent said
Run cases against the agent as deployed: the same model, system prompt, tool definitions, gateway and policy. A harness that calls the bare model tests the vendor's safety training, which is not what you ship. Replace only the far end of each tool with a sandboxed fixture that accepts the same calls and records what happened: a fake ERP with seeded vendors, a mail sink, a ledger you can inspect after the run.
Open tools supply the scaffolding. Microsoft's PyRIT is an open-source framework built to help security professionals and engineers proactively identify risks in generative AI systems [4]. NVIDIA's garak is an LLM vulnerability scanner built around probes, which generate interactions with a model, and detectors, which decide whether a response shows a given failure mode [5]. Both generate variants and drive conversations at volume. Neither knows what your agent's tools do, so the consequence check has to be yours. Run each case several times, because agent behavior is stochastic, and reset the fixtures before every run so one case's side effects cannot cause or mask the next.
The common scoring mistake is to search the reply for refusal language. That measures tone. An agent can say it cannot help after it has already called update_vendor_bank, and it can comply cheerfully with a request whose dangerous call was stopped downstream. Score each run on what reached the fixtures instead. No attempt: the agent never issued the call. Blocked: the call was issued and denied. Held: the call went to an approval and the approver, a person or the harness playing one, declined it. Effect: the prohibited change appears in the sandbox.
Only the last is a successful attack. The middle two are wins for different layers, and you want to know which layer won, because a defense that works only when a reviewer is paying attention is weaker than one that blocks. Weight results by consequence tier: revealing the system prompt and changing a vendor's bank account are both findings, and averaging them hides the one that matters.
Without a benign twin, the score rewards blocking everything
Every attack case needs a twin: a legitimate request that resembles the attack in vocabulary or shape and must succeed. If an attack hides a bank-change instruction in an invoice, the twin is a genuine bank change through the verified channel. If an attack asks the agent to email a payment file outside the company, the twin sends a remittance advice to the vendor's address on file. Without twins, the cheapest way to raise the score is to block everything that mentions bank details, and the program will quietly reward that.
Every run lands in one of four cells. Allowed attacks are misses to fix and blocked or held twins are false positives to tune; a program that watches only the attack row drifts toward blocking legitimate work.
Twins also expose defenses that react to words instead of authority. A rule that holds any message containing the phrase wire transfer will catch some attacks and hold every treasury analyst. The twin row shows that first, usually weeks before the complaints do.
Here is what a first run looks like on an accounts-payable agent at a fictional distributor, Corbel Supply. The white-text invoice footer telling the agent to change a vendor's bank details was caught: flagged at ingress, and update_vendor_bank asked regardless. But a case that split one large payment across several invoices under the review threshold was a miss, and the sandbox ledger showed the full amount paid to one vendor within an hour. The routine remittance twin was held too, a false positive the AP team would hit dozens of times a day; the fixes were a cumulative per-vendor limit and a known-destination rule for vendor addresses on file.
The strongest objection: a suite of known attacks finds only known attacks
The objection is fair. A regression suite reruns what someone already thought of, and an agent that passes it can still fall to a technique nobody wrote down. If the suite becomes the whole program, it measures last quarter's imagination.
The answer is in how the suite is built and fed, not in going back to point-in-time events. Cases are techniques with variants of encoding, language, placement and turn-splitting, so a fix has to hold against the technique, not one string. A held-out set that the people writing fixes never see keeps fixes from overfitting to the exact wording in the library. And the library has to grow: production incidents, near misses and newly published techniques should each add cases, because a library unchanged for a quarter measures last quarter's threat model. New attacks still have to be found; the suite is what makes sure each one stays fixed.
The objection does expose three ways suites quietly rot. The harness drifts to a staging model or a trimmed prompt to save money, so pin it to production versions and record them on every result. Scoring drifts back to refusals when a consequence check is hard to write for a new tool, so make that check part of the definition of done for giving the agent a tool. And sandboxes leak: a fixture that forwards to a real mail server turns a test into an incident, so make fixtures terminal and assert at the network layer that nothing leaves.
Findings become controls, and controls become regressions
Library cases run in a harness with sandboxed tools, results are scored by consequence, findings become controls, and fixed findings join the regression suite. Every model, prompt or policy change starts the loop again.
Each finding should end in a change with an owner: a policy rule, a tool permission, a threshold, a prompt fix, or a decision to accept the risk with a named owner and an expiry date. Prefer fixes that hold regardless of what the model decides. A prompt instruction to distrust invoice footers is worth having, but a rule that update_vendor_bank always asks still holds when the next model ignores the instruction. Track which findings were fixed only by prompt, and rerun those on every model change.
The fixed case joins the suite with its twin, and the suite runs on triggers: a model version change, a system prompt edit, a tool added or its description changed, an MCP server update, and any policy change. Run the full high-tier set on every trigger and sample the lower tiers. A regression run that takes hours gets skipped, so keep the blocking subset small enough to finish inside a deploy pipeline and run the long tail nightly.
Four numbers, each tagged with the versions it ran against
| Metric | Definition | Why it matters | Common trap |
|---|---|---|---|
| Attack success rate by tier | Share of attack runs ending in a prohibited effect, reported separately for each consequence tier | Shows whether the actions that can hurt you are protected | Averaging across tiers, so many low-tier wins hide one high-tier miss |
| Twin false-positive rate | Share of benign twin runs that were blocked or held when they should have run | Measures what the defenses cost real work | Counting an expected approval on a high-tier twin as a false positive |
| Time to fix | Days from a confirmed miss to a merged control that passes its regression case | Shows whether the program changes anything | Closing findings by editing the test instead of the control |
| Regression coverage | Share of library techniques that ran against the current model, prompt, tool and policy versions | Tells you whether the other numbers describe the deployed agent | Reporting results from runs on superseded versions |
Read the first two together, always. A falling attack success rate with a rising twin false-positive rate is not progress; the defenses are getting broader, not sharper, and the business will eventually route around them.
What the runtime layer gives the harness
Intertrace is the runtime layer a harness tests through, not a red-team tool. Its Go enforcer decides allow, flag, ask or block on every LLM, MCP and agent-to-agent call, and reads what a call does from the tool name and its arguments, so a payment or vendor bank change lands in the right consequence tier however the model phrases it. Every decision is a structured event recording the rule, layer and evidence, which gives a harness the blocked-or-held half of consequence scoring without parsing replies. Replay against the last 30 days of real decisions shows which outcomes a policy fix would move before it ships.
To start, pick one agent with a real write path, write ten cases against its top-tier tools with a twin for each, and run every case five times against the deployed configuration. If you cannot say which of the four outcomes each run ended in, the harness is not finished. If you can, you already know more about today's agent than last quarter's report does.
