Managing Intelligence: Skilled Experts and AI in Markets for Complex Products
In numerous high stakes markets skilled experts play a key role in facilitating consumer choice of complex products. New artificial intelligence (AI) technologies are increasingly being used to augment expert decisions. We study the role of technology and expertise in the market for health insurance, where consumer choices are widely known to be sub-optimal. Our analysis leverages the large-scale implementation of an AI-based decision support tool in a private Medicare exchange where consumers are randomized to skilled agents over time. We find that, prior to AI-based technology, skilled experts in this market exhibit the same type of inconsistent behavior found in previous studies of individual choices, costing consumers $1260 on average. The addition of AI-based decision support improves outcomes by $278 on average and substantially reduces heterogeneity in broker performance. Experts efficiently synthesize private information, incorporating AI-based recommendations along dimensions that are well suited to AI (e.g. total expected patient costs), but overruling AI-based recommendations along dimensions for which humans are better suited (e.g. specifics of doctor networks). As a result, switching plans, an ex-post measure of plan satisfaction, is meaningfully lower for agents making AI-based recommendations. While AI is a complement to skill on average, we find that it is a substitute across the skill distribution; lower quality agents provide better recommendations with AI than the top agents did without it. Overall productivity rises, with the introduction of decision support associated with a 21% reduction in call time for enrollment.
We thank Picwell for providing the data for this study. Ned Augenblick and Filip Matejka provided excellent comments. We thank participants at MIT, Northwestern, University of Arizona, BU/Harvard/MIT health seminar, University of Rochester, National University of Singapore, Western Economic Assoc. Conference, University of North Carolina, Haas, BYU and the NBER workshops on Economics of AI and Machine Learning in Health Care Markets for feedback. All authors hold ownership stakes in Picwell. The findings represent our own views and all errors are our own. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.
I would like to further disclose my financial relationship with Picwell Inc.. I hold shares in the company. There was no payment related to this research nor any direct financial support of the research project.Benjamin R. Handel
I would like to further disclose my financial relationship with Picwell Inc.. I hold shares in the company. There was no payment related to this research nor any direct financial support of the research project.Samuel H. Kina
I am an employee and share-holder of Picwell.Jonathan T. Kolstad
I would like to further disclose my financial relationship with Picwell Inc.. I hold shares in the company and have been compensated directly. There was no payment related to this research nor any direct financial support of the research project.