Prediction, Judgment and Uncertainty*
Ajay Agrawal, Joshua S. Gans and Avi Goldfarb
University of Toronto and NBER
Draft: 25th August 2017
We interpret recent developments in the field of artificial intelligence (AI) as
improvements in prediction technology. In this paper, we explore the consequences of
improved prediction in decision-making. To do so, we adapt existing models of
decision-making under uncertainty to account for the process of determining payoffs.
We label this process of determining the payoffs 'judgment.' There is a risky action,
whose payoff depends on the state, and a safe action with the same payoff in every
state. Judgment is costly; for each potential state, it requires thought on what the payoff
might be. Prediction and judgment are complements as long as judgment is not too
difficult. We next consider a tradeoff between prediction frequency and accuracy. We
show that as judgment improves, accuracy becomes more important relative to
frequency. We show that in complex environments with a large number of potential
states, the effect of improvements in prediction on the importance of judgment depend
a great deal on whether the improvements in prediction enable automated decisionmaking. We discuss the implications of improved prediction in the face of complexity
for automation, contracts, and firm boundaries.
Our thanks to Scott Stern, Hal Varian and participants at the AEA (Chicago), NBER Summer Institute (2017),
Harvard Business School, MIT, and University of Toronto for helpful comments. Responsibility for all errors remains
our own. The latest version of this paper is available at joshuagans.com.
We live in an age of paradox. Systems using artificial intelligence match or surpass human level performance in more and more domains, leveraging rapid advances in other technologies and driving soaring stock prices. Yet measured productivity growth has fallen in half over the past decade, and real income has stagnated since the late 1990s for a majority of Americans. Brynjolfsson, Rock, and Syverson describe four potential explanations for this clash of expectations and statistics: false hopes, mismeasurement, redistribution, and implementation lags. While a case can be made for each explanation, the researchers argue that lags are likely to be the biggest reason for paradox. The most impressive capabilities of AI, particularly those based on machine learning, have not yet diffused widely. More importantly, like other general purpose technologies, their full effects won't be realized until waves of complementary innovations are developed and implemented. The adjustment costs, organizational changes and new skills needed for successful AI can be modeled as a kind of intangible capital. A portion of the value of this intangible capital is already reflected in the market value of firms. However, most national statistics will fail to capture the full benefits of the new technologies and some may even have the wrong sign.
Artificial intelligence can use an individual's data to make predictions about what they might desire, be influenced by, or do. The use of an individual's data in this process raises privacy concerns. This article focuses on what is novel about the world of artificial intelligence and privacy, arguing that the chief novelty lies in the potential for data persistence, data repurposing, and data spillovers.
Is productivity growth inimical to employment? Canonical economic theory says no, but much recent economic theory says 'maybe' -- that is, rapid advances in machine capabilities may curtail aggregate labor demand as technology increasingly encroaches on human job tasks, ultimately immiserating labor. Autor and Salonmons refer to this immiseration scenario as the "robocalypse," and explore empirically whether it is coming to pass by analyzing the relationship between productivity growth and employment using country- and industry-level data for 19 countries over 35+ years. Consistent with both the popular ('robocalypse') narrative and the canonical Baumol hypothesis, they find that industry-level employment robustly falls as industry productivity rises, implying that technically progressive sectors tend to shrink. Simultaneously, the researchers show that country-level employment generally grows as aggregate productivity rises. Because sectoral productivity growth raises incomes, consumption, and hence aggregate employment, a plausible reconciliation of these results -- confirmed by the researchers' analysis -- is that the negative own-industry employment effect of rising productivity is more than offset by positive spillovers to the rest of the economy. Rapid productivity growth in primary and secondary industries has, however, generated a substantial reallocation of workers into tertiary services, which employs a disproportionate share of high-skill labor. In net, the sectoral bias of rising productivity has not diminished aggregate labor demand but has yielded skillbiased demand shifts.
In recent years, economists have revived the specter of slow growth and secular stagnation. From the point of view of economic history, what should we make of such doomster prophecies? As economic historians all know, for 97 percent of recorded history, the stationary state well describes the long-run dynamics of the world economy. Growth was slow, intermittent, and reversible. The Industrial Revolution rang in a period of sustained economic growth. Is that growth sustainable? One way to come to grips with that question is to analyze the brakes on economic growth before the Industrial Revolution and how they were released. Once these mechanisms are identified, we can look at the economic history of the past few decades and make an assessment of how likely growth is to continue. The answer Mokyr gives is simple: there is no technological reason for growth in economic welfare to slow down, although institutions may become in some areas a serious concern on the sustainability of growth.
Aghion, Jones, and Jones consider potential effects of artificial intelligence (A.I.) on economic growth. They start by modeling A.I. as a process where capital replaces labor at an increasing range of tasks and consider this perspective in light of the evidence to date. The researchers further discuss linkages between A.I. and growth as mediated by firm-level considerations, including organization and market structure. Finally, we engage the concepts of “singularities” and “superintelligence” that animate many discussions in the machine intelligence community. The goal throughout is to refine a set of critical questions about A.I. and economic growth and help shape an agenda for the field.
Athey provides an assessment of the early contributions of machine learning to economics, as well as predictions about its future contributions. They begin by briefly overviewing some themes from the literature on machine learning, and then draw some contrasts with traditional approaches to estimating the impact of counterfactual policies in economics. Next, Athey reviews some of the initial "off-the-shelf" applications of machine learning to economics, including applications in analyzing text and images. They then describe new types of questions that have been posed surrounding the application of machine learning to policy problems, including "prediction policy problems," as well as considerations of fairness and manipulability. Next, they briefly review of some of the emerging econometric literature combining machine learning and causal inference. Finally, Athey overviews a set of predictions about the future impact of machine learning on economics.
Artificial intelligence promises to improve existing goods and services, and, by enabling automation of many tasks, to greatly increase the efficiency with which they are produced. But it may have an even larger impact on the economy by serving as a new general-purpose "new method of invention" that can reshape the nature of the innovation process and the organization of R&D. This exploratory essay considers this possibility in three interrelated ways. First, Cockburn, Henderson, and Stern review the history of artificial intelligence, focusing in particular on the distinction between automation-oriented applications such as robotics and the potential for recent developments in "deep learning" to serve as a general-purpose method of invention. The researchers then assess preliminary evidence of this differential impact in changing nature of measurable innovation outputs in artificial intelligence, including papers and patents. They find strong evidence of a "shift" in the importance of application-oriented learning research since 2009 (relative to developments in robotics and symbolic systems research), and that a significant fraction of this upswing in application-oriented learning research was initially led by researchers outside the United States. Finally, Cockburn, Henderson, and Stern consider some of the implications of their findings, with a focus on both likely changes in the organization of the innovation process as well as the appropriate policy and institutional response that might be required if deep learning represents a meaningful general-purpose method of invention. From an organizational perspective, there is likely to be significant substitution away from more routinized labor-intensive research effort (often directed towards testing specific hypotheses in relatively small purpose-built datasets) towards research that takes advantage of the interplay between passively generated large datasets and enhanced prediction algorithms for phenomena that result from complex interdependencies. At the same time, the potential commercial reward is likely to usher in a period of racing, driven by powerful incentives for individual companies to acquire and control critical large datasets and application-specific algorithms. The researchers suggest that policies which encourage transparency and sharing of core datasets across both public and private actors can stimulate a higher level of innovation-oriented competition, and also allow for a higher level of research productivity going forward.
This paper was distributed as Working Paper 24449, where an updated version may be available.
Public Policy in an AI Economy
The Technological Elements of Artificial Intelligence
How Artificial Intelligence and Machine Learning Can Impact Market Design
The Impact of Artificial Intelligence on Innovation
AI and International Trade
Artificial Intelligence and Consumer Privacy
AI, Labor, Productivity and the Need for Firm-Level Data
AI as the next GPT: a Political-Economy Perspective
Prediction, Judgment and Complexity: A Theory of Decision Making and Artificial Intelligence
AI and Jobs: the role of demand
Artificial Intelligence and Its Implications for Income Distribution and Unemployment
Artificial Intelligence and Economic Growth
Finding Needles in Haystacks: Artificial Intelligence and Recombinant Growth
Artificial Intelligence and the Modern Productivity Paradox: A Clash of Expectations and Statistics