(In)Credible Development RCTs: False Discoveries, Low Power, and a Solution

We study the statistical credibility of randomized controlled trials (RCTs) in development economics and the optimal allocation of resources to replication. Using 745 principal-hypothesis tests from 190 articles in leading economics journals, we find that actual t-statistics peak well below the theoretical benchmark of 2.8 expected under 80 percent power, though above values associated with p-hacking. This suggests many RCTs are underpowered. Estimating an empirical Bayes mixture model that accounts for selective publication, we find that approximately 30 percent of published rejections are false discoveries, with average directional power at only 40 percent. At conventional significance thresholds, the posterior probability of a false discovery reaches 66 percent. Doubling original sample sizes lowers the average false discovery rate by five percentage points, whereas tightening the significance threshold to 0.005 yields a larger reduction at the cost of total discoveries. Finally, we characterize a funder’s optimal replication policy under a fixed budget and an evidence standard. Under standard regularity conditions, replications concentrate in an intermediate interval of original test statistics, and the optimal replication sample size is single-peaked in the original t-statistic. Illustrative calibrations show that successful replications reduce aggregate false discovery rates to 1.8–7.8 percent, depending on the evidence standard.