How Credible is the Credibility Revolution?
When economists analyze a well-conducted RCT or natural experiment and find a statistically significant effect, they conclude the null of no effect is unlikely to be true. But how frequently is this conclusion warranted? The answer depends on the proportion of tested nulls that are true and the power of the tests. I model the distribution of t-statistics in leading economics journals. Using my preferred model, 65% of narrowly rejected null hypotheses and 41% of all rejected null hypotheses with |t|<10 are likely to be false rejections. For the null to have only a .05 probability of being true requires a t of 5.48.
This research was supported in part by NSF grant SES-1851636. I am indebted to Qingyuan Chai and Xinze Liu for superb research assistance. This paper is based on my Presidential address to the Society of Labor Economists. I thank the participants there and at the Boston University empirical microeconomics workshop, the Hong Kong Baptist University Political Economy seminar, the Canadian Labour Economics Forum annual conference, and the (Australian) Labour Econometrics Workshop, and Isaiah Andrews, Henry Braun, and James MacKinnon for their helpful comments and questions. The usual caveat applies. The views expressed herein are those of the author and do not necessarily reflect the views of the National Bureau of Economic Research.