Prompt Injection Defense: A Practical Guide to Securing LLM Applications

Overview

A prompt injection occurs when untrusted text influences an AI system to disregard, reinterpret, or bypass the instructions that developers intended it to follow. The untrusted text may come from a user message, a retrieved document, an email, a web page, a file, or the output of another model.

The central security assumption is simple: an LLM can process instructions and data in the same context, but it does not reliably enforce the authority boundary between them. A sentence inside a retrieved document may look like an instruction to the model even when it should be treated only as content. For that reason, secure LLM applications should not depend on a system prompt alone to protect secrets, authorize actions, or enforce business rules.

Threat modeling should cover five related risks:

• Direct prompt injection: a user attempts to override the application’s intended behavior.
• Indirect prompt injection: instructions are hidden in content the application retrieves, browses, summarizes, or processes.
• Tool abuse: the model requests an unsafe action, uses a tool outside its purpose, or chains ordinary tools into a harmful result.
• Data exfiltration: sensitive instructions, credentials, personal data, or internal content are exposed through an answer or tool call.
• Privilege escalation: a low-trust user or document causes the application to perform an action reserved for a more trusted identity.

For broader implementation context, pair this checklist with a practical comparison of AI developer tools and a RAG tutorial focused on building and maintaining retrieval workflows.

Checklist by scenario

1. User-facing chat and assistants

• Define what the assistant may answer, refuse, retrieve, and change.
• Keep secrets, credentials, internal prompts, and authorization decisions outside the model’s accessible context whenever possible.
• Treat user-provided instructions as requests, not as permission to change application policy.
• Validate structured output against a schema before it reaches downstream code.
• Apply ordinary application authentication and authorization before executing an action requested by the model.
• Log relevant requests, tool calls, validation failures, and approvals without storing unnecessary sensitive content.

A useful design pattern is to separate the model’s response from the application’s decision. The model can propose an action, but deterministic code should check identity, scope, parameters, and approval requirements before execution.

2. Retrieval-augmented generation and document search

• Classify documents by source, owner, sensitivity, and trust level before indexing them.
• Preserve document metadata so the application can apply access controls during retrieval, not only after generation.
• Tell the model explicitly that retrieved text is reference material and may contain irrelevant or hostile instructions.
• Limit retrieved content to the minimum needed for the task.
• Filter, quarantine, or review documents that contain suspicious instruction patterns when the use case warrants it.
• Test whether a malicious passage can cause the model to reveal system context, alter an answer, or call a tool.

Retrieval quality and security are connected. A system that retrieves excessive, stale, or poorly scoped content gives an attacker more opportunity to influence the model. The RAG evaluation guide can help teams assess retrieval quality, grounded answers, and failure cases as separate concerns.

3. Browsing, email, and file-processing agents

• Assume web pages, emails, attachments, and office documents are untrusted inputs.
• Separate content extraction from action planning. Reading an instruction should not authorize the agent to follow it.
• Restrict network access, file paths, domains, and data sources to an explicit allowlist where practical.
• Block access to credentials, environment files, private keys, and unrelated user data.
• Require confirmation before sending messages, publishing content, deleting records, making purchases, or changing permissions.
• Render external content safely and avoid passing raw markup or executable content into tools without validation.

For high-impact workflows, use a two-stage process: first produce a proposed action with evidence and parameters; then have deterministic code or an authorized person approve and execute it.

4. Tool-enabled agents and workflow automation

• Give each tool the narrowest possible purpose and parameter set.
• Use separate credentials and service identities for different tools and environments.
• Enforce authorization at the tool or API layer rather than trusting the model to apply permissions.
• Validate URLs, file names, query filters, record identifiers, and numeric limits before execution.
• Set time, cost, recursion, rate, and transaction limits for multi-step workflows.
• Record the user, model request, selected tool, arguments, result, and approval state for later review.

Least privilege matters even when a model is usually well behaved. A compromised or confused agent should not automatically inherit broad database, filesystem, messaging, or administrative access.

5. Systems handling sensitive information

• Map which inputs, retrieved sources, prompts, outputs, logs, and tool results can contain sensitive data.
• Minimize data sent to the model and redact information that is not required for the task.
• Define retention and access rules for prompts, responses, traces, and evaluation datasets.
• Test for cross-user data exposure, especially when using shared retrieval indexes, caches, or conversation memory.
• Design a safe failure mode that withholds content rather than guessing when authorization or provenance is unclear.

What to double-check

Use the following comparison when choosing controls. No single measure reliably stops every form of prompt injection; layered controls reduce the consequences of a successful manipulation.

Double-check the boundary between model output and executable code. Do not interpolate unvalidated model text into SQL, shell commands, browser actions, templates, access-control decisions, or API requests. Use parameterized interfaces, strict schemas, and normal security controls.

Also review prompt and workflow changes systematically. A prompt evaluation framework and prompt optimization workflow can be extended with security cases, not just quality and accuracy scores.

Reusable test-case template

For each test, record:

1. Entry point: chat, retrieval, email, browser, file, memory, or tool result.
2. Attacker capability: what content or request can the tester control?
3. Protected asset: secret, document, account, transaction, system instruction, or service.
4. Expected behavior: refuse, summarize safely, ask for approval, or continue with restricted access.
5. Observed behavior: include the model response, tool calls, and validation outcomes.
6. Impact and fix: describe the worst plausible result and the control that should address it.

Common mistakes

• Relying on a hidden prompt as a security boundary. Treat prompts as behavior guidance, not as a substitute for authorization, isolation, or secret management.
• Trusting retrieved content because it came from an internal source. Internal documents can be outdated, incorrectly permissioned, edited by users, or intentionally compromised.
• Giving an agent broad tools for convenience. Narrow tools often require more integration work, but they reduce the consequences of ambiguous or malicious requests.
• Checking only the final answer. A harmless-looking answer may follow a dangerous hidden tool call, expose data through metadata, or trigger an external side effect.
• Testing only obvious jailbreak language. Include indirect instructions, encoded text, poisoned documents, conflicting sources, long context, and multi-step attempts.
• Logging everything without a data policy. Security logs can become a second source of sensitive-data exposure. Define access, retention, redaction, and review procedures.
• Assuming model upgrades preserve security behavior. A change in model, provider, retrieval method, tool schema, or context window can alter both failure modes and test results.

When to revisit

Review prompt injection defenses before launching a new LLM feature and whenever the application’s trust boundaries change. In particular, repeat the review when you add a tool, connect a new retrieval source, enable browsing, introduce memory, change model providers, modify system instructions, or expand the agent’s permissions.

Include security cases in regular release testing and before seasonal planning cycles or high-volume campaigns. Recheck workflows when teams change the documents, APIs, prompts, or approval rules that support them. A content automation workflow, for example, should be retested if it gains publishing, email, or file-management capabilities; the AI content workflow automation guide offers useful process context.

After an incident or near miss, preserve the relevant trace, identify the first untrusted input, determine which control failed, and update both the application and its test corpus. Do not stop at rewriting the prompt if the underlying issue is excessive privilege or missing validation.

Before your next deployment, confirm:

1. Untrusted content is labeled and handled as data.
2. Every tool has narrow permissions, validated parameters, and an independent authorization check.
3. Sensitive data is minimized, isolated, and protected in logs and retrieval indexes.
4. Consequential actions require an appropriate approval or deterministic policy check.
5. Direct, indirect, multi-step, and data-exfiltration test cases have expected outcomes.
6. Monitoring, rollback, and incident-response owners are documented.

Secure LLM applications by controlling what the model can see, what it can request, and what the surrounding software will actually execute. Revisit those controls whenever inputs, tools, permissions, or workflows change.