Skip to content
Search

Qurtoo

Papers

Short digests of research papers about prompting and agents: what each paper found, why it matters, and how to use it.

  • Development

    Measuring AI Ability to Complete Long Software Tasks

    This paper measures AI ability with a 50% task-completion time horizon: how long a task takes a skilled human, for tasks the AI completes half the time. When it was written, frontier models were at about 50 minutes on the authors' software tasks, and the horizon had been doubling roughly every seven months since 2019.

    Thomas Kwa, Ben West, Joel Becker, et al.

  • Development

    SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

    SWE-agent showed that coding agents do better with tools designed for them. Its agent-computer interface gives the model purpose-built commands to navigate a repository, view and edit files, and run tests. It reached 12.5% pass@1 on SWE-bench and 87.7% on HumanEvalFix, which the authors reported as state of the art.

    John Yang, Carlos E. Jimenez, Alexander Wettig, et al.

  • Development

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

    Benchmark that tests whether an LLM can take a real GitHub issue and a full repository and produce a passing code change. Went from 2% solve rate in 2023 to 70%+ by 2025 -- the clearest quantitative record of agentic coding progress we have.

    Carlos E. Jimenez, John Yang, Alexander Wettig, et al.

  • Development

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    This paper tested whether a strong model can grade other models' answers the way people would. GPT-4 as a judge agreed with human preferences over 80% of the time, about as often as humans agree with each other. It also named the judge's biases: favoring an answer for its position, for its length, and for being its own.

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al.

  • Development

    LoRA: Low-Rank Adaptation of Large Language Models

    Fine-tune a giant model by training tiny adapter matrices alongside it, leaving the base frozen. Cuts training memory by 10-100x, lets you host one base model with many LoRAs for different tasks, runs on a single consumer GPU.

    Edward J. Hu, Yelong Shen, Phillip Wallis, et al.