NATIONAL BUREAU OF ECONOMIC RESEARCH
NATIONAL BUREAU OF ECONOMIC RESEARCH
loading...

Split-Sample Strategies for Avoiding False Discoveries

Michael L. Anderson, Jeremy Magruder

NBER Working Paper No. 23544
Issued in June 2017
NBER Program(s):Development Economics Program, Labor Studies Program, Public Economics Program

Preanalysis plans (PAPs) have become an important tool for limiting false discoveries in field experiments. We evaluate the properties of an alternate approach which splits the data into two samples: An exploratory sample and a confirmation sample. When hypotheses are homogeneous, we describe an improved split-sample approach that achieves 90% of the rejections of the optimal PAP without requiring preregistration or constraints on specification search in the exploratory sample. When hypotheses are heterogeneous in priors or intrinsic interest, we find that a hybrid approach which prespecifies hypotheses with high weights and priors and uses a split-sample approach to test additional hypotheses can have power gains over any pure PAP. We assess this approach using the community-driven development (CDD) application from Casey et al. (2012) and find that the use of a hybrid split-sample approach would have generated qualitatively different conclusions.

download in pdf format
   (645 K)

email paper

Machine-readable bibliographic record - MARC, RIS, BibTeX

Document Object Identifier (DOI): 10.3386/w23544

Users who downloaded this paper also downloaded* these:
Bhagwati Export Performance and Other Effects
Fafchamps and Labonne w21842 Using Split Samples to Improve Inference about Causal Effects
Engen and Skinner w5826 Taxation and Economic Growth
Aguiar, Bils, Charles, and Hurst w23552 Leisure Luxuries and the Labor Supply of Young Men
Schwert w9277 Anomalies and Market Efficiency
 
Publications
Activities
Meetings
NBER Videos
Themes
Data
People
About

National Bureau of Economic Research, 1050 Massachusetts Ave., Cambridge, MA 02138; 617-868-3900; email: info@nber.org

Contact Us