Skip to content
Search
AgenticPapers

Reflexion: Language Agents with Verbal Reinforcement Learning

Authors: Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao

Published March 20, 2023

Read the paper: arXiv 2303.11366 · PDF

Why it matters

Reflexion is an early, widely cited version of the critique-and-retry loop found in many agentic workflows. Instead of changing the model's weights, it turns feedback into words: a failed test, an error message or a score becomes a short note on what went wrong, and that note is kept in an episodic memory the agent reads on its next attempt.

The feedback can be a number or free text, and it can come from outside (a compiler or test run) or from the model's own evaluation. That makes the pattern easy to reuse. The paper reports significant gains over a baseline agent across sequential decision-making, coding and language reasoning tasks.

How to use it

Give your agent a real feedback signal first. Tests, a validator or a checklist work better than asking the model whether it is done. After a failed attempt, ask the model to write two or three sentences on why it failed and what it will change, keep those notes in agent memory for the next attempt, and cap the number of retries. See reflection for the general prompt pattern and ReAct for the loop Reflexion builds on.

Abstract

Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.