Defending Against Indirect Prompt Injection Attacks With Spotlighting
Authors: Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, Emre Kiciman
Published March 20, 2024
Read the paper: arXiv 2403.14720 · PDF
Why it matters
A model reads one stream of text. When an app pastes a user's request and a fetched document into the same prompt, the model has no reliable way to know which part came from whom, and that is the gap indirect prompt injection uses. Spotlighting gives the model a continuous signal of where each piece of text came from, by transforming the untrusted input before it goes in.
The paper tests three versions: delimiting (wrapping the untrusted text in marked boundaries), datamarking (interleaving a special character through the text) and encoding (transforming the text, for example to base64). It is a prompt-level defense, so it is cheap to add to an existing app. The authors recommend encoding only with high-capacity models, since an earlier-generation model they tested did worse on encoded text. The numbers come from the authors' own experiments with GPT-family models, so measure it on your model and your data before you rely on it.
How to use it
Before you put a web page, email or uploaded file into the prompt, transform it and say so in the system prompt. For datamarking, replace the spaces in the untrusted text with a marker character and tell the model that text containing that marker is data to read, never instructions to follow. After you add it, check task quality on your own model and data, since the paper's results come from GPT-family models. Treat spotlighting as one layer of prompt injection defense, alongside limits on what tools the agent can call.
Abstract
Large Language Models (LLMs), while powerful, are built and trained to process a single text input. In common applications, multiple inputs can be processed by concatenating them together into a single stream of text. However, the LLM is unable to distinguish which sections of prompt belong to various input sources. Indirect prompt injection attacks take advantage of this vulnerability by embedding adversarial instructions into untrusted data being processed alongside user commands. Often, the LLM will mistake the adversarial instructions as user commands to be followed, creating a security vulnerability in the larger system. We introduce spotlighting, a family of prompt engineering techniques that can be used to improve LLMs' ability to distinguish among multiple sources of input. The key insight is to utilize transformations of an input to provide a reliable and continuous signal of its provenance. We evaluate spotlighting as a defense against indirect prompt injection attacks, and find that it is a robust defense that has minimal detrimental impact to underlying NLP tasks. Using GPT-family models, we find that spotlighting reduces the attack success rate from greater than 50% to below 2% in our experiments with minimal impact on task efficacy.
Articles that use this
- Red Teaming Your AI Agents With Prompt Chains Before Attackers DoSecurity
- Red-Teaming Your Own AI App: A Prompt Injection Test Suite You Can Run TodaySecurity
- Building Prompt Guardrails: Input Filtering and Output Validation for LLM-Powered AppsSecurity
- Prompt Patterns That Catch MCP Tool Poisoning Before Your Agent Executes ItSecurity
- Prompt Injection Testing for Production Apps: A Red Team PlaybookSecurity

