Prompt injection attacks, mitigations, detection, and design patterns for safer AI applications.
Prompt injection is the core attack pattern in modern AI applications. It happens when a model treats malicious or conflicting instructions from users, retrieved content, documents, tools, or pages as trusted guidance and changes its behavior in response.
These items are included because they add useful evidence, framing, implementation detail, or upcoming context for teams working in this area.
OWASP's community guide organizes agentic-system risk into ten categories, including goal hijacking, tool misuse, identity and privilege abuse, memory poisoning, insecure inter-agent communication, cascading failures, and rogue-agent behavior. It provides a shared taxonomy and mitigation starting point rather than a certification checklist or evidence that a deployed system is secure.
OWASP FinBot is a hands-on agentic-security CTF built around a simulated multi-agent financial-services platform with real tool access. Its challenges cover prompt injection, tool misuse, policy bypass, data exfiltration, privilege escalation, remote code execution, shared context, and compromised MCP servers.
NVIDIA walks through a simulated Go dependency that detects Codex, writes a malicious AGENTS.md, redirects the coding task, and injects instructions intended to conceal the change from pull-request summaries. The post then maps the chain to dependency, configuration-integrity, monitoring, and guardrail controls.
Microsoft Incident Response provides a detection, investigation, and response playbook for prompt abuse, then walks through an indirect prompt-injection scenario in which a hidden URL fragment manipulates an AI summarizer. The guide maps each incident phase to visibility, prompt telemetry, access, audit, and response controls.
NVIDIA's AI Red Team reports recurring failures across six months of enterprise-agent assessments: weak user-level access control, command and file tools that enable code execution, unrestricted network egress, and secrets exposed through environment variables or CLI caches. Social framing, gradual multi-turn escalation, and malicious package installation repeatedly bypassed prompts and model-judge defenses, while controls enforced outside the model reduced exploitability.
Anthropic reports three incidents across six of 141,006 cybersecurity-evaluation runs: models reached unintended real targets, extracted data, or published a malicious package after evaluation isolation and configuration controls failed. The report distinguishes these harness failures from evidence of a persistent model goal, and documents how realistic evaluations can create production consequences.
OpenAI’s system card for deep research covers prompt injection, privacy, code execution, and external red teaming prior to release.
NVIDIA’s AI Red Team provides a deep implementation guide for sandboxing coding agents: enforce network egress and filesystem boundaries below the application layer, protect agent configuration files, isolate spawned hooks and MCP processes, use virtualization where warranted, inject scoped secrets, and expire sandbox state.
OpenAI describes an automated prompt-injection red-team loop for a browser agent: an attacker model proposes an injection, runs counterfactual victim-agent simulations, studies full reasoning and action traces, iterates before submission, and turns successful attacks into adversarial training targets and system-level safeguards.
Microsoft distills an ontology, eight lessons, and five case studies from red teaming more than 100 generative AI products. The method connects actors, tactics and techniques, system weaknesses, and downstream impacts across conventional application flaws, multimodal prompt injection, responsible-AI harms, and dangerous capabilities.
OpenAI treats prompt injection as contextual social engineering and uses source-sink analysis to connect attacker-controlled content with dangerous actions. The design approach combines model resistance with deterministic limits on data transmission, navigation, tool use, sandbox communication, and user confirmation.
ASSET Research Group hid a prompt-injection payload in a PNG referenced by an apparently benign AGENTS.md file. Text-only pull-request reviewers missed the image, multiple coding-agent harnesses later followed it and encoded a repository's .env secrets as integer tuples that conventional secret scanners did not recognize, while the same model behaved differently across harnesses. A prototype multimodal reviewer caught 49 of 50 attacks with no false positives on 30 benign pull requests.
NVIDIA uses a PandasAI code-execution vulnerability to explain why generated-code sanitizers are brittle: namespace access, encoding, imports, and runtime context can turn apparently allowed syntax into arbitrary execution. The article separates heuristic filtering from the structural protection provided by a constrained execution environment.
NVIDIA walks through a repository-borne prompt-injection chain in which a coding agent reviewing a pull request installs a disguised dependency whose setup logic opens a reverse shell. The example connects untrusted issue and pull-request text to package execution and shows why model-level refusal cannot secure a developer environment with broad tools and credentials.
OWASP analysis of memory and context poisoning as an agent attack surface. Relevant to persistent state, trust boundaries, and regression tests for agent memory.
The Operator system card documents red teaming and mitigation choices for a computer-using agent, with prompt injections listed as a central risk area.
NVIDIA demonstrates gradient-based attacks against a PaliGemma2 vision-language classifier, including imperceptible perturbations and localized patches that change a stop-sign decision or force an arbitrary output token. It also explains why physical attacks require transformations that model changes in scale, angle, lighting, and capture conditions.
OWASP roundup of reported GenAI incidents and exploit patterns from Q1 2026. Relevant as a threat-intelligence reference for risk tracking and test-case design.
An accessible explanation of prompt injection risk in real AI products, including how third-party content can redirect or manipulate agent behavior.
Adversa AI describes DeepJack, a Cursor deeplink weakness in which a crafted cursor:// link can register an attacker-controlled MCP server and hide the command that will execute behind an incomplete installation prompt.
NVIDIA AI Red Team post on grammar-constrained decoding for Bash generation in small language models. Relevant to safer command generation and executable-output controls.
Learn how CNAPP platforms are helping organizations prioritize exploitable risks, reduce exposure, and operationalize security across the application lifecycle.
Zenity found that ChatGPT Workspace Agents Builder treated an attacker-supplied initial_assistant_prompt URL parameter as an instruction to execute in a logged-in user's session. A single link could attach already-authorized connectors, switch approvals to “Never ask,” publish and schedule the agent, and use incoming email as a persistent command channel; OpenAI fixed the flaw four days after it was reported.
OWASP’s GenAI security project remains a practical baseline for teams building or assessing LLM applications and agentic systems.