Experimental evidence on the productivity effects of generative artificial intelligence
Authors: Shakked Noy, Whitney Zhang
Published July 14, 2023
Read the paper: DOI 10.1126/science.adh2586
Why it matters
This is controlled evidence, published in Science, on the kind of writing many office jobs involve, with each task matched to the participant's occupation. Random assignment means the gains can be attributed to access to the chatbot, and the tasks were incentivized so people had a reason to do them well.
Besides the 40% time saving and 18% quality gain, inequality between workers decreased. People who used ChatGPT in the experiment were twice as likely to report using it in their real job two weeks later, and 1.6 times as likely two months later.
How to use it
Use it as evidence when you decide where to start with AI at work: midlevel writing is where this study saw clear gains. The tasks were set assignments in an online experiment, so check how well they match your own work before you expect the same numbers. Build in a review step for facts and figures, since speed gains are only worth having if the claims in the draft hold up. For tasks with more judgment in them, read Navigating the Jagged Technological Frontier, and for a habit that cuts made-up facts in drafts, see Chain-of-Verification.
Abstract
We examined the productivity effects of a generative artificial intelligence (AI) technology, the assistive chatbot ChatGPT, in the context of midlevel professional writing tasks. In a preregistered online experiment, we assigned occupation-specific, incentivized writing tasks to 453 college-educated professionals and randomly exposed half of them to ChatGPT. Our results show that ChatGPT substantially raised productivity: The average time taken decreased by 40% and output quality rose by 18%. Inequality between workers decreased, and concern and excitement about AI temporarily rose. Workers exposed to ChatGPT during the experiment were 2 times as likely to report using it in their real job 2 weeks after the experiment and 1.6 times as likely 2 months after the experiment.

