Skip to content
Search
SecurityPapers

The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

Authors: Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, Alex Beutel

Published April 19, 2024

Read the paper: arXiv 2404.13208 · PDF

Why it matters

The paper names a root cause of prompt injection: models often treat a developer's system prompt and text from untrusted users or third parties as if they carried the same authority. The instruction hierarchy fixes an order of trust and trains the model on generated examples where it should follow instructions that agree with higher-priority ones and ignore lower-priority instructions that conflict with them.

The authors built it as a research prototype by fine-tuning GPT-3.5 Turbo, and they describe existing models as lacking this capability. It explains what a trust order between instructions can and can't do for you. Any protection comes from how a model is trained. The paper reports greater robustness without claiming it is complete, and the ordering only helps if your app puts each piece of text in the right place.

How to use it

You can't apply the training method yourself unless you train models, but you can lay out your app to match the trust order it describes. Put your rules in the system prompt. Pass web pages, files and function call results as tool or data content, never pasted into the system or user message where they borrow higher trust. Keep code-level checks on anything destructive, since a trained preference can still fail. Pair it with architectural defenses such as the ones in CaMeL.

Abstract

Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts. In this work, we argue that one of the primary vulnerabilities underlying these attacks is that LLMs often consider system prompts (e.g., text from an application developer) to be the same priority as text from untrusted users and third parties. To address this, we propose an instruction hierarchy that explicitly defines how models should behave when instructions of different priorities conflict. We then propose a data generation method to demonstrate this hierarchical instruction following behavior, which teaches LLMs to selectively ignore lower-privileged instructions. We apply this method to GPT-3.5, showing that it drastically increases robustness -- even for attack types not seen during training -- while imposing minimal degradations on standard capabilities.