The Economics of Scale-Up
Most randomized controlled trials (RCT) of social programs test interventions at modest scale. While the hope is that promising programs will be scaled up, we have few successful examples of this scale-up process in practice. Ideally we would like to know which programs will work at large scale before we invest the resources to take them to scale. But it would seem that the only way to tell whether a program works at scale is to test it at scale. Our goal in this paper is to propose a way out of this Catch-22. We first develop a simple model that helps clarify the type of scale-up challenge for which our method is most relevant. Most social programs rely on labor as a key input (teachers, nurses, social workers, etc.). We know people vary greatly in their skill at these jobs. So social programs, like firms, confront a search problem in the labor market that can lead to inelastically-supplied human capital. The result is that as programs scale, either average costs must increase if program quality is to be held constant, or else program quality will decline if average costs are held fixed. Our proposed method for reducing the costs of estimating program impacts at large scale combines the fact that hiring inherently involves ranking inputs with the most powerful element of the social science toolkit: randomization. We show that it is possible to operate a program at modest scale n but learn about the input supply curves facing the firm at much larger scale (S × n) by randomly sampling the inputs the provider would have hired if they operated at scale (S × n). We build a simple two-period model of social-program decision making and use a model of Bayesian learning to develop heuristics for when scale-up experiments of the sort we propose are likely to be particularly valuable. We also present a series of results to illustrate the method, including one application to a real-world tutoring program that highlights an interesting observation: The noisier the program provider’s prediction of input quality, the less pronounced is the scale-up problem.
This paper was made possible by the generous support of the Abdul Latif Jameel Poverty Action Lab (J-PAL), Chicago Beyond, the Polk Bros Foundation, and the William T. Grant Foundation. For vital assistance in making this intervention possible we thank Barbara Algarin, Rukiya Curvey-Johnson, Antonio Gutierrez, Alan Safran and the staff of the Chicago Public Schools and SAGA Innovations, as well as the Crown Family Foundation, the Lloyd A. Fry Foundation, and the IMC Charitable Foundation. For help accessing administrative data we thank the Chicago Public Schools, Jeffrey Broom, Sarah Dickson, Kylie Klein, and Stacy Norris. For outstanding help with project fieldwork and data assembly and analysis, we thank Valentine Gilbert, Rowan Gledhill, Nathan Hess, Zachary Honoroff, Julia Quinn, Catherine Schwarz, Hannah Shaw, Maitreyi Sistla, and John Wolf. For his excellent assistance with the development of the translation exercises we thank Juchi Pratt. For their very helpful suggestions and comments we thank Esther Duflo, Joe Hotz, John List, Sendhil Mullainathan, Jesse Rothstein, Jesse Shapiro, Don and Liz Thompson, and seminar participants at Duke University and the 2016 FEW conference. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research, or any other funder or data provider. All opinions and any errors are our own.