Prompt injection defense is a design and testing problem, not a single instruction added to a system prompt. This guide provides a reusable checklist for finding injection paths, separating trusted instructions from untrusted content, limiting tool access, validating outputs, and deciding when a human must approve an action.
Overview
A prompt injection occurs when text supplied to an LLM attempts to change the application’s intended behavior. The text may come directly from a user, or indirectly from a web page, document, email, database record, retrieved knowledge chunk, tool response, or generated intermediate result. The attack does not need to look like a conventional hacking payload. It can be a natural-language request such as “ignore previous instructions,” a hidden instruction in a document, or content designed to make the model reveal data or call a tool.
Jailbreaks are related but broader: they try to make a model bypass behavioral constraints or produce responses the application is meant to prevent. In an LLM application, both problems should be handled as untrusted-input and authorization risks. A model can help interpret a request, but it should not be the final authority for access control, payment approval, secret retrieval, deletion, or other consequential operations.
Start by mapping the application’s trust boundaries. Record which inputs are system-controlled, developer-controlled, user-controlled, retrieved, or returned by tools. Then identify where data is combined with instructions, where the model can invoke tools, and where model output becomes an instruction for another component. This map is more useful than relying on prompt wording alone.
For a broader prompt testing process, see the Prompt Engineering QA Workflow. If the application handles personal or confidential information, pair this work with the LLM Data Privacy Checklist.
Checklist by scenario
1. Direct user prompts
- Define the allowed task. State what the assistant may do, what it must refuse, and which actions require confirmation.
- Separate request interpretation from authorization. Let the model identify a requested operation, but verify identity, permissions, account ownership, and limits in application code.
- Assume the user can see and manipulate the conversation. Never treat a user-supplied claim such as “I am an administrator” as proof of identity or privilege.
- Limit sensitive context. Do not place secrets, unrelated customer records, internal policies, or hidden credentials in the model context merely because they might be useful.
- Test instruction-conflict prompts. Include requests to reveal system messages, change the assistant’s role, bypass a refusal, expose hidden context, or perform an action without confirmation.
2. Retrieval-augmented generation
- Label retrieved content as data. Delimit documents clearly and tell the model that instructions found inside them are not application instructions.
- Filter before retrieval. Apply access controls, tenant filters, document status checks, and metadata validation before content reaches the model.
- Filter after retrieval. Inspect chunks for unexpected instruction-like content, secrets, malicious links, or irrelevant material before including them in context.
- Preserve provenance. Keep document identifiers and source boundaries so the application can explain which content influenced an answer.
- Test indirect injections. Place adversarial instructions in titles, footers, comments, PDFs, HTML, code blocks, and ordinary-looking business documents.
3. Tool calling and external actions
- Use an allowlist. Expose only the tools required for the task, with narrowly defined arguments and scopes.
- Validate every argument outside the model. Check types, ranges, identifiers, destinations, ownership, and business rules in deterministic code.
- Separate read and write capabilities. A tool that searches records should not automatically be able to edit or delete them.
- Require approval for high-impact actions. Payments, account changes, external messages, destructive operations, and disclosure of sensitive information should have an explicit approval point.
- Use time and rate limits. Limit repeated calls, recursive tool use, bulk exports, and requests that could create unexpected cost or load.
- Return minimal tool results. A tool should provide only the fields needed for the next step, not an entire database row or unfiltered API response.
4. Multi-step agents and workflows
- Give each step a narrow contract. Define the input schema, output schema, permitted tools, and failure behavior for every agent or workflow stage.
- Prevent instruction laundering. Treat model-generated plans, summaries, and tool responses as untrusted data when they move between stages.
- Set a maximum execution budget. Bound the number of steps, tool calls, tokens, and elapsed time.
- Log decisions and transitions. Capture the request, retrieved source identifiers, tool arguments, approvals, validation failures, and final action without unnecessarily storing sensitive content.
| Attack pattern | Typical goal | Primary control |
|---|---|---|
| Direct override | Replace the application’s task or reveal hidden instructions | Clear trust boundaries, limited context, refusal tests |
| Indirect injection | Make retrieved or tool-returned content control the assistant | Content labeling, retrieval filtering, provenance, output checks |
| Tool manipulation | Invoke an unauthorized action or alter arguments | Allowlisted tools, server-side authorization, approval gates |
| Data exfiltration | Expose secrets or information from another user or system | Least-privilege context, tenant isolation, redaction, access checks |
| Jailbreak or role shift | Bypass behavioral or domain restrictions | Safe defaults, constrained workflows, adversarial regression tests |
What to double-check
Instruction hierarchy is not authorization. System and developer messages can establish intended behavior, but they do not make an unsafe action safe. A model may still misunderstand, follow a maliciously framed request, or produce a persuasive but incorrect tool call. Enforce permissions and business rules in code.
Output formatting is not output safety. Valid JSON can contain a dangerous URL, an unauthorized account identifier, or an instruction that another component executes. Parse structured output, validate each field, and reject values outside an explicit schema. If output is rendered in a browser, apply the appropriate encoding and sanitization for that context.
Retrieval quality is also a security concern. Check whether a user can retrieve the source, whether results cross tenant boundaries, and whether stale permissions remain in indexes. A well-written answer based on the wrong document is still a security failure.
Canaries should be harmless and traceable. Add synthetic markers to test documents or records so a red-team run can show whether content escaped its intended boundary. Do not use real secrets as canaries. Record where the marker appeared, which step exposed it, and whether any tool call followed.
Observability must support investigation. At minimum, retain request and workflow identifiers, model and application versions, tool names, validation outcomes, approval decisions, and relevant source identifiers. Apply retention and access rules appropriate to the data.
Common mistakes
- Putting a warning in the prompt and stopping there. Warnings can help, but they do not replace permissions, validation, isolation, or monitoring.
- Giving an agent broad tools for convenience. Start with read-only, narrowly scoped capabilities and add access only when a documented use case requires it.
- Trusting retrieved documents because they came from an internal source. Internal content can be stale, compromised, incorrectly permissioned, or written by someone who did not expect it to control an AI workflow.
- Testing only obvious attack phrases. Vary language, formatting, encoding, languages, document locations, multi-turn conversations, and combinations of benign and malicious instructions.
- Logging everything without a data plan. Complete transcripts may contain sensitive information. Define what is needed for debugging, who can access it, and how long it is retained.
- Failing open after a validation error. If a tool argument is missing, ambiguous, or outside policy, stop or request clarification rather than silently substituting a value.
- Changing the prompt without regression testing. Prompt edits, model changes, SDK upgrades, new retrieval sources, and new tools can alter security behavior. Keep representative tests and compare results before release.
A practical red-team test template can use this structure:
- Scenario: Describe the user task and the assets or actions at risk.
- Entry point: Record whether the payload enters through chat, a document, retrieval, a tool response, or another workflow step.
- Payload variation: Test direct instructions, indirect instructions, role changes, encoded text, misleading context, and multi-turn escalation.
- Expected behavior: Define the safe response, permitted tools, required approval, and data that must not be disclosed.
- Observed behavior: Capture the model output, tool arguments, validation result, and downstream effect.
- Fix and regression case: Document the control added and retain the test for future releases.
When to revisit
Re-run this checklist before adding a tool, retrieval source, agent step, or external integration. Revisit it after changing system prompts, permissions, output schemas, model providers, SDKs, safety settings, or workflow orchestration. A model or framework update can change tool-calling behavior and how the application interprets edge cases, so repeat representative tests rather than assuming earlier results still apply.
Schedule a recurring review at least around major release and planning cycles, and include a focused red-team exercise when the application’s users, data types, or consequences change. Review direct prompts, indirect retrieval attacks, data leakage, unauthorized tool calls, denial-of-service patterns, and failure handling. Compare results with the previous review and assign an owner to every unresolved finding.
Use this short release gate:
- Map every new input and trust boundary.
- Confirm least-privilege access for context and tools.
- Run direct and indirect prompt injection tests.
- Validate tool arguments and structured outputs outside the model.
- Exercise approval gates and failure paths.
- Check tenant isolation, logging, retention, and redaction.
- Record model, prompt, SDK, and retrieval changes.
- Keep a regression test for each fixed vulnerability.
Prompt injection protection is strongest when it is treated like ordinary application security: define assets, constrain authority, validate at boundaries, test realistic abuse cases, and make changes reviewable. For operational reliability alongside security controls, the AI API Error Handling Guide covers retry, fallback, and validation patterns that help prevent an uncertain model response from becoming an uncontrolled action.