Predicting Experimental Results: Who Knows What?

Stefano DellaVigna, Devin Pope

NBER Working Paper No. 22566
Issued in August 2016
NBER Program(s):Development Economics, Economics of Education, Health Care, Labor Studies, Public Economics, Productivity, Innovation, and Entrepreneurship

Academic experts frequently recommend policies and treatments. But how well do they anticipate the impact of different treatments? And how do their predictions compare to the predictions of non-experts? We analyze how 208 experts forecast the results of 15 treatments involving monetary and non-monetary motivators in a real-effort task. We compare these forecasts to those made by PhD students and non-experts: undergraduates, MBAs, and an online sample. We document seven main results. First, the average forecast of experts predicts quite well the experimental results. Second, there is a strong wisdom-of-crowds effect: the average forecast outperforms 96 percent of individual forecasts. Third, correlates of expertise---citations, academic rank, field, and contextual experience--do not improve forecasting accuracy. Fourth, experts as a group do better than non-experts, but not if accuracy is defined as rank ordering treatments. Fifth, measures of effort, confidence, and revealed ability are predictive of forecast accuracy to some extent, especially for non-experts. Sixth, using these measures we identify `superforecasters' among the non-experts who outperform the experts out of sample. Seventh, we document that these results on forecasting accuracy surprise the forecasters themselves. We present a simple model that organizes several of these results and we stress the implications for the collection of forecasts of future experimental results.

download in pdf format
   (2238 K)

email paper

Machine-readable bibliographic record - MARC, RIS, BibTeX

Document Object Identifier (DOI): 10.3386/w22566

Published: Stefano DellaVigna & Devin Pope, 2018. "Predicting Experimental Results: Who Knows What?," Journal of Political Economy, vol 126(6), pages 2410-2456.

Users who downloaded this paper also downloaded* these:
DellaVigna and Pope w22193 What Motivates Effort? Evidence and Expert Forecasts
Cravino and Levchenko w22498 Multinational Firms and International Business Cycle Transmission
DellaVigna, List, Malmendier, and Rao w22043 Estimating Social Preferences and Gift Exchange at Work
Graham w22575 Identifying and Estimating Neighborhood Effects
Deaton and Cartwright w22595 Understanding and Misunderstanding Randomized Controlled Trials
NBER Videos

National Bureau of Economic Research, 1050 Massachusetts Ave., Cambridge, MA 02138; 617-868-3900; email:

Contact Us