Measuring the Productivity Impact of Generative AI

Featured in print Digest
This is a scatter plot titled AI Assistance and Customer Complaint Resolutions. The y-axis is labeled Complaint resolutions per hour and ranges from negative 0.2 to 0.8 The x-axis is labeled months before and after AI deployment and ranges from negative 10 to 5. There is a dashed vertical line at 0.  All data points to the left of the dashed vertical line at 0 hover close to a 0 value. At 0, the value is about 0.3 complaint resolutions per hour. This value increases to 0.6 at 3 months after deployment before leveling out to just below 0.6 The note line reads Thin bars represent 95% confidence intervals The source line reads Source: Researchers’ calculations using data from customer support agents provided by a Fortune 500 enterprise software company


Customer support agents using an AI tool to guide their conversations saw a nearly 14 percent increase in productivity, with 35 percent improvements for the lowest skilled and least experienced workers, and zero or small negative effects on the most experienced/most able workers,Erik BrynjolfssonDanielle Li, and Lindsey R. Raymond report in Generative AI at Work (NBER Working Paper 31161).

Using call data from roughly 5,000 agents working for a Fortune 500 software company, the researchers tracked the duration, quality, and outcome of customer support interactions as the company introduced a Generative Pre-trained Transformer (GPT) AI tool. The tool was rolled out to the agents gradually, mostly between November 2020 and February 2021. For a control group, the researchers also collected data from agents who did not receive the tool over 2020 and 2021. The AI tool was intended to support the work of human customer support agents, offering them potential responses to customer queries. The agents could choose to take those suggestions or ignore them and enter their own responses.

With AI assistance, customer service agents could handle more calls per hour and increase their resolution rate.

The researchers find that customer support agents utilizing the AI tool increased the number of customer issues resolved per hour by 13.8 percent. They attribute the increase to three factors: agents, who could participate in multiple chats at once, spent about 9 percent less time per chat, handled about 14 percent more chats per hour, and successfully resolved about 1.3 percent more chats overall. Measures of customer satisfaction showed no significant change, suggesting that the productivity improvements did not come at the expense of interaction quality.

The researchers divide the data by agents’ length of tenure and pre-AI productivity, and find that the benefits of using the AI tool were greatest among less experienced and lower skill workers, who saw gains of 35 percent, with little to no negative effects on top performing/most experienced workers. An agent using the AI tool who had just two months’ tenure at the firm performed as well as an agent with six months’ tenure working without the tool. The researchers suggest that newer and lower skilled workers may have more to learn than higher skilled and more established workers, and that AI tools can help them adopt the skills and behavior of more experienced workers more quickly. Text analysis of agents’ conversations supports this interpretation.

All agents changed their communication patterns after beginning to use the AI tool, but the change among lower-performing agents was greater. This may be because the AI tool based its suggestions on the work style and outputs of the company’s most productive agents, and therefore spread their pattern of behavior to newer and less skilled workers. For instance, the developers of the AI tool found that top performers were able to determine the underlying technical issue, based on a customer’s description, twice as fast as lower performers. The AI tool, trained using the best examples of resolved queries, learned to connect specific query phrases to useful diagnostic questions and potential solutions. The AI tool was also able to give more frequent feedback than a human manager. This gave new hires and lower performers the opportunity to improve faster than they would have without the tool, iterating with each call rather than only following managerial reviews.

The researchers also noticed that customers were more likely to express positive sentiments, and less likely to request help from a supervisor, when interacting with agents using AI assistance than when interacting with those who were not. Perhaps reflecting the improved tenor of the exchanges, attrition rates among agents with access to the AI tool were 8.6 percent lower than the comparable rates for agents without such access.

Emma Salomon