Adversaries may input malicious prompts directly into an LLM’s user interface.[1] The prompts can be designed to bypass safeguards, manipulate context, override previous instructions, or coerce the LLM into unauthorized behavior.[2]
Adversaries may use direct prompt injection to Generate Content prohibited by the LLM's guardrails or to induce the system to utilize its capabilities in unintended or prohibited ways.[3][4] Adversaries may achieve this by instructing the LLM to ignore previous instructions or safety rules, or by impersonating a role such as a researcher or privileged user.[5]
| ID | Name | Description |
|---|---|---|
| C0064 | AI-Assisted Breach of Mexico's Government Infrastructure |
During the AI-Assisted Breach of Mexico's Government Infrastructure, the adversary abused Claude Code by utilizing a file-write operation within the prompt that created a |
| C0062 | Anthropic AI-orchestrated Campaign |
During the Anthropic AI-orchestrated Campaign, the adversary entered a prompt that claimed the user was an employee of a cybersecurity firm and that Claude was being used for defensive cybersecurity testing to authorize restricted operations.[7] |
| ID | Mitigation | Description |
|---|---|---|
| M1038 | Execution Prevention |
Prevent user-supplied AI prompts from directly authorizing consequential actions. Enforce tool, API, command, and resource authorization outside the model and reject requested actions that exceed the permissions associated with the initiating user or agent task. |
| M1026 | Privileged Account Management |
Restrict access to privileged AI capabilities and require explicit authorization or human approval for AI-generated actions involving administrative functions, sensitive resources, or privileged tools. |
| M1018 | User Account Management |
Associate AI interactions with authenticated identities and constrain AI-mediated actions to the permissions of the initiating user. Do not allow prompt content to establish or elevate the authorization context used by downstream tools or services. |
| M1017 | User Training |
Train users responsible for AI-enabled systems to recognize attempts to manipulate system instructions, role boundaries, or safety controls through user-provided prompts and to independently validate consequential AI-generated actions. |