When Scale and Replication Work: Learning from Summer Youth Employment Experiments
Because successful human capital interventions often fail to scale or replicate, public investment decisions require understanding how program size, context, and implementation shape program effects. This paper uses two new randomized controlled trials of summer youth employment programs in Chicago and Philadelphia to demonstrate how multiple experiments can help explain replicability and inform the expansion of promising approaches. Even when these programs grow or change models across contexts, participation consistently reduces criminal justice involvement. It may also decrease the need for child protective services and behavioral health treatment. Experimental variation in program model and local provider generates no detectable heterogeneity, suggesting that effects replicate partly because variability in implementation does not matter. There is, however, individual-level heterogeneity that explains differences in effect magnitudes across populations and informs optimal targeting; youth at higher risk of socially costly outcomes experience larger benefits. Identifying more interventions that combine this pattern of treatment heterogeneity with robust replicability could aid efforts to reduce social inequality efficiently.
This project was supported by Award No. 2016-R2-CX-0049, awarded by the National Institute of Justice, Office of Justice Programs, U.S. Department of Justice, a State and Local Innovation Initiative grant (Philadelphia) and Social Policy Research Initiative grant (Chicago) from J-PAL North America, the Robert R. McCormick Foundation, and Project Development Grant Program funding from Poverty Solutions at the University of Michigan. Louise Geraghty, Brenda Mathias, Matt Repka, Misuzu Schexnider, and Lauren Shaw provided highly-accomplished project management; Kalen Flynn managed the Philadelphia qualitative data collection and analysis; Raquel Chavez, Kenny Hofmeister, Angela Hsu, Owen McCarthy, and Mary Clair Turner provided excellent research assistance; Marianne Bertrand provided invaluable support to the Chicago experiment, as did Greg Ridgeway for the Philadelphia study. The author thanks Jon Davis, Brian Jacob, and Basit Zafar for extremely helpful comments. We are grateful to the Chicago Department of Family and Support Services, the Philadelphia Youth Network, Inc., the Philadelphia mayor's office, and the University of Chicago Urban Labs for their partnership on these projects. We also thank the City of Philadelphia, the Philadelphia Police Department, the School District of Philadelphia, the Chicago Police Department, and the Chicago Public Schools for graciously allowing the use of their administrative data. Any further use of the data is subject to approval of each agency. The opinions, findings, and conclusions or recommendations expressed in this publication are those of the author and do not necessarily reflect those of these organizations. The studies are registered in the American Economic Association Registry under trial numbers 2451 (WorkReady) and 805 (OSC+). The views expressed herein are those of the author and do not necessarily reflect the views of the National Bureau of Economic Research.
Sara B. Heller, 2022. "When scale and replication work: Learning from summer youth employment experiments," Journal of Public Economics, vol 209.