Everyone in the room believed the support agent could not refund more than $150. The payments key said otherwise. It could refund any amount on any order, and the limit existed only as a sentence in the system prompt. That finding, from the fictional retailer this guide follows, surfaced in the second of five passes, and no amount of jailbreak research would have found it.
Most agent threat models end up in one of two states. Nobody writes one because the system feels too new to model, or someone writes forty pages on jailbreak taxonomies that never mention which credential the agent holds. Both miss the same thing. An agent's risk is mostly set by what it is allowed to do and who can influence it, and both are knowable from configuration you already have. Model those, for one agent, in one afternoon, on one page.
Before you start: one agent, four people, the real configuration
Adam Shostack's four-question frame is still the best scaffold: “What are we working on? What can go wrong? What are we going to do about it? Did we do a good job?” [1]. The five passes answer the first two, the one-page model the third, and the tests at the end the fourth. Keep the scope to one agent; a platform-wide model rarely finishes.
The example is a support agent at Larkspur Goods, a fictional mid-size online retailer. It reads incoming helpdesk tickets, looks up orders, issues refunds up to a limit (assume $150), and emails customers with the outcome. It runs on a hosted model API and handles a few thousand tickets a week. Nothing about it is exotic, which is the point: this shape of agent is being deployed everywhere.
Book four people for four hours: the agent's owner (here, the support operations lead), the engineer who wired its tools, a security engineer to facilitate, and someone who can say what the business will accept. Have them bring the system prompt, tool definitions, credentials and their scopes, and twenty real transcripts, including a few the team considers odd.
Pass 1 (45 minutes): trust follows who wrote the text, not where it is stored
Draw every input and output, then mark the trust boundaries. For an agent, a boundary exists wherever text from someone other than the operator can reach the model's context, and wherever the agent's output leaves your control.
The customer ticket crosses a trust boundary on the way in and the outbound email crosses one on the way out; both are warning paths because the model connects them. Order lookup, refunds and the model API sit behind their own boundaries.
Larkspur's first draft had one untrusted arrow: the ticket. The transcripts added two more. Order records include a free-text delivery note the customer typed at checkout, and the ticket thread includes quoted replies from earlier emails. Both arrive through internal systems, so the team had labeled them trusted. They are not. Text a customer wrote is customer text even after it sits in your database for a month.
Mark the model API as a boundary too. Everything in the context window, including order details from tool results, leaves your network on every call. That is a question for your provider contract; put it on the diagram so nobody forgets it.
Pass 2 (45 minutes): compare what each credential permits with what was intended
Authority is what the credentials and tools permit, not what the system prompt asks. For each tool, write down the credential behind it, the scope the designers intended and the scope the credential really has. The gap between the last two is usually the best finding of the afternoon. OWASP's Excessive Agency entry names its three causes: excessive functionality, excessive permissions and excessive autonomy [3].
| Tool | Credential | Intended scope | Actual scope found | Gap |
|---|---|---|---|---|
| Ticket read | Helpdesk API token | Read the current ticket | Read and write every ticket, including other agents' queues | Write access and cross-queue reads are unnecessary |
| Order lookup | Database service account | Orders for the ticket's customer | Read-only on all orders and customer profiles | No binding between ticket requester and order queried |
| Refunds | Payments API key | Refunds up to $150 per order | Refunds of any amount on any order; the limit exists only in the system prompt | Ceiling is enforced by the model's compliance, not by a control |
| Transactional email API key | Reply to the ticket requester | Send from the company domain to any address | Recipient is not bound to the requester | |
| Ticket close | Helpdesk API token (same) | Close resolved tickets | Close, merge or delete any ticket | Delete was never needed |
Write the refund row down plainly, and record anything enforced only by instructions to the model as unenforced.
Pass 3 (40 minutes): rank actions by what the call does, not what the tool is called
Consequence is the worst realistic outcome of one action, judged by reversibility and reach. Sort each action into five tiers, classifying by what the call does, arguments included. A tool called update_ticket that can also delete is a delete tool.
Reads and reversible changes can run, irreversible changes ask in production, and anything outbound, financial or administrative always asks. The tier sets the default; any exception should be a recorded decision.
At Larkspur, order lookup is a read, an internal note is a reversible change and deleting a ticket is irreversible. Emailing a customer is outbound and a refund is money, so two of the agent's four core jobs sit at the top of the ladder. The business wants small refunds to run without a person. That is a legitimate choice if a named owner accepts it and the payments layer enforces the ceiling per order and per day, outside the model.
If a tier is empty, say so on the page. It tells the next reviewer that adding an administrative action is a change worth re-modeling.
Pass 4 (40 minutes): list every writer, then pair each with the strongest tool in reach
Indirect prompt injection happens when a model takes in external content, such as a web page or a file, that carries instructions [2]. The working question is simpler: who can put text in front of this agent? List writers, not systems.
- Anyone who can email the support address writes the ticket subject, body and attachments, including PDFs and images parsed into text.
- Any customer writes the delivery note and the name and address fields on their order, which the lookup tool returns verbatim.
- Previous correspondents write the quoted history in the thread, which may include forwarded mail from third parties.
- Support staff and contractors write the macros and knowledge-base articles the agent retrieves; check who can edit them.
- The model provider writes every response, and anyone who changes a tool definition or connected server writes the tool descriptions the model reads.
Now pair each writer with the highest-tier tool reachable in the same session. At Larkspur, an anonymous ticket author can reach refunds and outbound email in one conversation. That pairing is the core risk: untrusted input, customer data and an outbound channel, joined by a model that cannot reliably tell instructions from content. OWASP recommends segregating and labeling external content and keeping privileged functions under application control [2]. Both help. Assume they will sometimes fail.
Pass 5 (50 minutes): if the answer is "the prompt says not to", nothing stops it
Run the last pass twice. First assume the model is mistaken: it picks the wrong order, retries a refund after a timeout, or emails a customer with a similar name. Then assume the worst writer from Pass 4 steers it completely. Walk each scenario call by call and ask what stops it. If the honest answer is the prompt, the scenario is uncontrolled.
STRIDE adapts well when each category is reframed around the agent, and MITRE ATLAS, a public knowledge base of adversary tactics and techniques targeting AI systems, helps name techniques and find case studies [4].
| Category | Agent question | Larkspur finding |
|---|---|---|
| Spoofing | Whose identity does the agent accept, and from where? | The agent trusts a customer name typed in the ticket body over the verified sender address. |
| Tampering | Which inputs can change the agent's plan? | Instructions in a delivery note were followed in one transcript during testing. |
| Repudiation | Can you show which input caused a given action? | Refund logs record the amount and order, but not the ticket or message that triggered them. |
| Information disclosure | Can one customer's data reach another person? | Order lookup is not bound to the requester, and email can go to any address. |
| Denial of service | What happens if the agent loops or is flooded? | No per-ticket cap on tool calls; a reopen loop could resend emails and burn model spend. |
| Elevation of privilege | Can input make the agent exceed its intended authority? | A ticket can request a refund above $150, and nothing outside the prompt refuses it. |
The hostile trace that mattered most: a ticket claims a damaged parcel and carries hidden text asking the agent to refund a different order in full and send the order history to an outside address. The agent holds every permission needed for both. Two controls break the chain: bind order lookups and refunds to the verified requester, and bind outbound email to the ticket's requester, with any other recipient requiring a person.
Two habits derail this pass. The room drifts into which jailbreak works on which model; bring it back to authority, because a perfect jailbreak against an agent with no consequential tools is a minor finding. And fixes get recorded as prompt edits. A prompt change can make a failure rarer, but it is not a control and does not close a risk.
The page: ranked risks, each with a person and a date
Spend the last 20 minutes on the page: the agent's name, owner, model and version date; the data-flow diagram; the authority table; the ranked risks; accepted risks with the name of who accepted them; and the triggers for re-modeling, such as a new tool, credential or input channel. This lines up with the NIST AI RMF, where MAP establishes the context for framing risk and GOVERN covers the roles and accountability for managing it [5].
| Rank | Risk | Tier | Fix | Owner |
|---|---|---|---|---|
| 1 | Injected ticket triggers a refund above the limit | Money | Enforce per-order and daily ceilings in the payments layer; scope the key to refunds only | Payments engineering lead |
| 2 | Customer data emailed to a third party | Outbound | Bind recipient to the verified requester; anything else asks a person | Support platform engineer |
| 3 | Cross-customer order disclosure | Read | Bind lookups to the requester's customer ID at the tool, not in the prompt | Support platform engineer |
| 4 | Ticket deletion by a confused or steered agent | Irreversible | Remove delete from the token's scope | Helpdesk administrator |
| 5 | No trail from triggering input to action | Cross-cutting | Log the ticket and message ID with every tool call | Security engineering |
Every risk gets a person, not a team, and a date; risks owned by security engineering in general stay open. Tie re-modeling to the triggers on the page, so that adding an email tool to a read-only agent forces a new pass.
Did we do a good job? Rerun these after every change
Shostack's fourth question deserves its own artifact [1]: turn each ranked risk into a test you rerun whenever the prompt, model or tools change.
- Submit a ticket asking for a full refund of an unrelated $400 order. Expected: the payments layer refuses, whatever the model does.
- Call the refund tool directly with $151, bypassing the model. Expected: refusal, which proves the limit lives outside the prompt.
- Ask the agent to email the order confirmation to an address other than the requester's. Expected: it waits for a person or is refused.
- Place an order whose delivery note contains instructions, then open a ticket about it. Expected: the instructions are ignored and the attempt is logged with its source.
- Simulate a payments timeout and let the agent retry. Expected: one refund, enforced by an idempotency key.
- Take the policy or approval service down. Expected: consequential actions fail closed.
- Pick any refund from the logs and trace it to the exact ticket message that caused it in under five minutes.
Some of these fixes can live in a runtime gateway rather than in each tool. Intertrace sits inline between the agent and its model, tools and MCP servers, reads what each call does from the tool name and its arguments, and assigns one of five consequence tiers; outbound messages, payments and admin changes always ask a person unless a named, expiring exception covers them. Each decision is recorded with its rule, layer and evidence, which closes the repudiation gap, and the enforcer fails closed. The payments ceiling and the customer-ID binding remain changes your engineers make in the tools.
