Research Registries: Facts, Myths, and Possible Improvements
The past few decades have ushered in an experimental revolution in economics whereby scholars are now much more likely to generate their own data. While there are virtues associated with this movement, there are concomitant difficulties. Several scientific disciplines, including economics, have launched research registries in an effort to attenuate key inferential issues. This study assesses registries both empirically and theoretically, with a special focus on the AEA registry. We find that over 90% of randomized control trials (RCTs) in economics do not register, only 50% of the RCTs that register do so before the intervention begins, and the majority of these preregistrations are not detailed enough to significantly aid inference. Our empirical analysis further shows that using other scientific registries as aspirational examples is misguided, as their perceived success in tackling the main issues is largely a myth. In light of these facts, we advance a simple economic model to explore potential improvements. A key insight from the model is that removal of the (current) option to register completed RCTs could increase the fraction of trials that register. We also argue that linking IRB applications to registrations could further increase registry effectiveness.
We thank Ariel Listo and Xinyi Hong for excellent research assistance. Discussions with Vittorio Bassi, Eszter Czibor, Stefano DellaVigna, Michael Kremer, Min Sok Lee, Shengwu Li, Ulrike Malmendier, Gautum Rao, and Sutanuka Roy significantly improved the paper. We also thank participants of the UChicago Experimental Economics Seminar for their feedback and the Becker Friedman Institute for financial support. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.