Skip to content
Search
AppliedPapers

Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality

Authors: Fabrizio Dell'Acqua, Edward McFowland III, Ethan R. Mollick, Hila Lifshitz-Assaf, Katherine Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, Karim R. Lakhani

Read the paper: DOI 10.2139/ssrn.4573321

Why it matters

The paper gives a name to something most people who use AI at work have noticed: what AI does well and badly doesn't follow a neat line. Two tasks in the same workflow, of about the same difficulty, can sit on opposite sides of that jagged frontier. The authors measured it with real consultants on realistic consulting work, randomly assigned to no AI, AI, or AI with a short prompting overview.

The headline gains were large on the 18 tasks inside the frontier. The warning matters just as much: on the task outside it, AI users did worse than people working without AI. Using AI everywhere is a risk when you can't tell which side of the frontier a task is on.

How to use it

Before you hand a recurring task to AI, test it on a few cases where you already know the right answer, and check the results yourself. Keep a short list of tasks where AI helped and tasks where it misled you, and update it as tools change. On complex judgment calls that mix several sources, do your own reasoning first and use AI to check it, and watch for hallucination. The source is an SSRN working paper; read it alongside Generative AI at Work and Noy and Zhang's writing experiment.

Abstract

We introduce and study the concept of a “jagged technology frontier” to describe the uneven impact of artificial intelligence (AI) capabilities, where AI assistance improves performance for some tasks but worsens it for others, even within the same knowledge workflow and with a seemingly similar level of difficulty. In collaboration with the global management consulting firm Boston Consulting Group, we have developed realistic management consulting tasks and examined the human performance implications of using AI to perform complex and knowledge-intensive work. The preregistered experiment involved 758 knowledge workers. After establishing a performance baseline on similar tasks, subjects were randomly assigned to one of three conditions: no AI access, GPT-4 AI access, or GPT-4 AI access with a prompt engineering overview. For each one of a set of 18 realistic knowledge tasks within the frontier of AI capabilities ranging from creative to analytical tasks, subjects using AI outperformed those not using AI, completing 12.2% more tasks and completing them 25.1% more quickly on average while also delivering solutions of significantly improved quality. However, for a complex managerial task selected to be outside the frontier, subjects using AI were 19% less likely to produce correct solutions compared with those without AI, pointing to potential limitations of AI supporting knowledge workers. We discuss the positive and negative implications of AI-aided human performance in knowledge-intensive tasks.