Split-Sample Strategies for Avoiding False Discoveries
Preanalysis plans (PAPs) have become an important tool for limiting false discoveries in field experiments. We evaluate the properties of an alternate approach which splits the data into two samples: An exploratory sample and a confirmation sample. When hypotheses are homogeneous, we describe an improved split-sample approach that achieves 90% of the rejections of the optimal PAP without requiring preregistration or constraints on specification search in the exploratory sample. When hypotheses are heterogeneous in priors or intrinsic interest, we find that a hybrid approach which prespecifies hypotheses with high weights and priors and uses a split-sample approach to test additional hypotheses can have power gains over any pure PAP. We assess this approach using the community-driven development (CDD) application from Casey et al. (2012) and find that the use of a hybrid split-sample approach would have generated qualitatively different conclusions.
Michael L. Anderson is Associate Professor, Department of Agricultural and Resource Economics, University of California, Berkeley, CA 94720 (E-mail: firstname.lastname@example.org). Jeremy Magruder is Associate Professor, Department of Agricultural and Resource Economics, University of California, Berkeley, CA 94720 (E-mail: email@example.com). They gratefully acknowledge funding from the NSF under Award 1461491, “Improved Methodologies for Field Experiments: Maximizing Statistical Power While Promoting Replication,” approved in September 2015. They thank Katherine Casey, Ted Miguel, Sendhil Mullainathan, Ben Olken, and conference and seminar participants at the Univ of Washington, Stanford, UC Berkeley, and Notre Dame for insightful comments and suggestions and are grateful to Aluma Dembo and Elizabeth Ramirez for excellent research assistance. All mistakes are the authors’. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.