SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Authors: John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, Ofir Press
Published May 6, 2024
Read the paper: arXiv 2405.15793 · PDF
Why it matters
The paper treats the language model as a new kind of user with its own needs, the way an IDE is built around a human programmer. The interface has search commands, a file viewer that shows up to 100 lines at a time, and an edit command whose changes are rejected if a linter finds a syntax error. With the same model, the authors compared it with a shell-only agent and removed its parts one at a time, and the design of the interface changed how well the model performed. That makes the harness, meaning the commands and feedback around the model, something you can engineer and measure.
It follows directly from SWE-bench, by an overlapping team. Where SWE-bench defined the test, SWE-agent showed one way to build an agent that does well on it, and why the interface matters.
How to use it
When you build an agent with tool use, design its tools for a model reader. Give it a few clear commands instead of a raw shell, return short and focused output instead of whole files, and make errors explicit so the model can recover. Change one interface detail at a time and rerun your evals, since the paper's point is that these details move results. The scores are from 2024, so use the paper for its method and check current results before you quote a number.
Abstract
Language model (LM) agents are increasingly being used to automate complicated tasks in digital environments. Just as humans benefit from powerful software applications, such as integrated development environments, for complex tasks like software engineering, we posit that LM agents represent a new category of end users with their own needs and abilities, and would benefit from specially-built interfaces to the software they use. We investigate how interface design affects the performance of language model agents. As a result of this exploration, we introduce SWE-agent: a system that facilitates LM agents to autonomously use computers to solve software engineering tasks. SWE-agent's custom agent-computer interface (ACI) significantly enhances an agent's ability to create and edit code files, navigate entire repositories, and execute tests and other programs. We evaluate SWE-agent on SWE-bench and HumanEvalFix, achieving state-of-the-art performance on both with a pass@1 rate of 12.5% and 87.7%, respectively, far exceeding the previous state-of-the-art achieved with non-interactive LMs. Finally, we provide insight on how the design of the ACI can impact agents' behavior and performance.

