Using a Satisficing Model of Experimenter Decision-Making to Guide Finite-Sample Inference for Compromised Experiments
This paper presents a simple decision-theoretic economic approach for analyzing social experiments with compromised random assignment protocols that are only partially documented. We model administratively constrained experimenters who satisfice in seeking covariate balance. We develop design-based small-sample hypothesis tests that use worst-case (least favorable) randomization null distributions. Our approach accommodates a variety of compromised experiments, including imperfectly documented re-randomization designs. To make our analysis concrete, we focus much of our discussion on the influential Perry Preschool Project. We reexamine previous estimates of program effectiveness using our methods. The choice of how to model reassignment vitally affects inference.
We thank Juan Pantano and Azeem Shaikh for comments on early drafts of this paper. We thank Cheryl Polk and Lawrence Schweinhart of the HighScope Educational Research Foundation for their assistance in data acquisition, sharing historical documentation, and their longstanding partnership with the Center for the Economics of Human Development. This research was supported in part by: the Buffett Early Childhood Fund; NIH Grants R01AG042390, R01AG05334301, and R37HD065072; and the American Bar Foundation. The views expressed in this paper are solely those of the authors and do not necessarily represent those of the funders, the official views of the National Institutes of Health, nor the views of the National Bureau of Economic Research.
James J Heckman & Ganesh Karapakula, 2021. "Using a satisficing model of experimenter decision-making to guide finite-sample inference for compromised experiments [Sampling-based versus design-based uncertainty in regression analysis]," The Econometrics Journal, Royal Economic Society, vol. 24(2), pages 1-39. citation courtesy of