Generative AI at Work
Authors: Erik Brynjolfsson, Danielle Li, Lindsey Raymond
Published April 23, 2023
Read the paper: arXiv 2304.11771 · PDF
Why it matters
This study measured AI assistance on live customer support work at a real company, with thousands of agents. The assistant was introduced to different groups at different times, which let the authors compare agents with and without access. The average gain was 15% more issues resolved per hour, and the spread matters as much as the average: newer and lower-skilled agents improved both speed and quality, and the most skilled agents gained little.
The authors also found signs that the assistant helped agents learn and improved English fluency, especially among international agents, and that gains were largest on rarer problems where agents had less experience. Customers were more polite and less likely to ask for a manager.
How to use it
If you are bringing AI into a support or service team, start with newer staff and with the rare, unfamiliar cases, which is where this study saw the biggest gains. Measure output and quality by experience level instead of only the team average, since the most experienced people may see small quality drops. Keep in mind that this is one company, one task type and one assistant, so read 15% as what that team saw and measure your own. For deciding which tasks to hand over, read it with Navigating the Jagged Technological Frontier.
Abstract
We study the staggered introduction of a generative AI-based conversational assistant using data from 5,172 customer support agents. Access to AI assistance increases worker productivity, as measured by issues resolved per hour, by 15% on average, with substantial heterogeneity across workers. Less experienced and lower-skilled workers improve both the speed and quality of their output while the most experienced and highest-skilled workers see small gains in speed and small declines in quality. We also find evidence that AI assistance facilitates worker learning and improves English fluency, particularly among international agents. While AI systems improve with more training data, we find that the gains from AI adoption are largest for relatively rare problems, where human agents have less baseline training and experience. Finally, we provide evidence that AI assistance improves the experience of work along two key dimensions: customers are more polite and less likely to ask to speak to a manager.

