SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Authors: Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan

Published October 10, 2023

Read the paper: arXiv 2310.06770 · PDF

Why it matters

SWE-bench is why the narrative of "AI can write code" shifted from demo to production between 2023 and 2025. Every coding assistant vendor reports their number on it; the progress curve is the fastest in any LLM benchmark family.

More importantly, it catalyzed the design of coding-specific agent scaffolds: file search, repo navigation, test-run feedback loops. The paper defines the problem; the last 2 years of agent architecture answers it.

How to use it

Read if you're building coding agents or evaluating them. The SWE-bench Lite and SWE-bench Verified subsets are the practical entry points for your own eval work. Use their harness as a reference for how to structure an agent eval with real test feedback.

Abstract

We introduce SWE-bench, an evaluation framework consisting of 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories.