Skip to content
Search

Qurtoo

Papers

Short digests of research papers about prompting and agents: what each paper found, why it matters, and how to use it.

  • Security

    Defeating Prompt Injections by Design

    CaMeL is a prompt injection defense built into the agent's architecture. The plan comes only from the user's trusted request, so untrusted data can't change what the agent does, and security policies are checked whenever a tool is called. It solved 77% of AgentDojo tasks with provable security, against 84% for an undefended agent.

    Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, et al.

  • Security

    The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

    The instruction hierarchy is a training method that teaches a model to rank instructions by where they come from, so the developer's system prompt outranks the user and both outrank third-party text such as tool results. Applied to GPT-3.5, it made the model much more robust, including against attack types it hadn't been trained on, with little loss on ordinary tasks.

    Eric Wallace, Kai Xiao, Reimar Leike, et al.

  • Security

    Defending Against Indirect Prompt Injection Attacks With Spotlighting

    Spotlighting is a family of prompt techniques that mark untrusted text so the model can tell it apart from the user's instructions. In the authors' tests with GPT-family models, it cut the indirect prompt injection success rate from over 50% to under 2%, with little effect on the task itself.

    Keegan Hines, Gary Lopez, Matthew Hall, et al.