In Praise of Moderation: Suggestions for the Scope and Use of Pre-Analysis Plans for RCTs in Economics
Pre-Analysis Plans (PAPs) for randomized evaluations are becoming increasingly common in Economics, but their definition remains unclear and their practical applications therefore vary widely. Based on our collective experiences as researchers and editors, we articulate a set of principles for the ex-ante scope and ex-post use of PAPs. We argue that the key benefits of a PAP can usually be realized by completing the registration fields in the AEA RCT Registry. Specific cases where more detail may be warranted include when subgroup analysis is expected to be particularly important, or a party to the study has a vested interest. However, a strong norm for more detailed pre-specification can be detrimental to knowledge creation when implementing field experiments in the real world. An ex-post requirement of strict adherence to pre-specified plans, or the discounting of non-pre-specified work, may mean that some experiments do not take place, or that interesting observations and new theories are not explored and reported. Rather, we recommend that the final research paper be written and judged as a distinct object from the “results of the PAP”; to emphasize this distinction, researchers could consider producing a short, publicly available report (the “populated PAP”) that populates the PAP to the extent possible and briefly discusses any barriers to doing so.
The authors thank Madison Cann, Elizabeth Cao, Laura Feeney, Sarah Gault, Kalila Jackson-Spieker, James Turitto, and Keesler Welch for their support in writing this article, and Mary Ann Bates, Shawn Cole, Jeremy Magruder, Edward Miguel, William Parienté, and the J-PAL Board for helpful comments on an earlier draft. Two of the authors of this piece, Esther Duflo and Lawrence Katz, were on the AEA committee that created the Registry and are on the committee that currently provides oversight for the Registry. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.
Lawrence F. Katz
For the sake, of full disclosure, I list my positions as an officer, director, or board member of non-profit organizations:
(a) Editor, Quarterly Journal of Economics
(b) Board of Directors, MDRC
(c) Advisory Board, Opportunity and Inclusive Growth Institute, Federal Reserve Bank of
(d) Co-Scientific Director, J-PAL North AmericaAnja Sautmann
Anja Sautmann is overseeing the operation and maintenance of the AEA RCT registry as part of her role at the Abdul Latif Jameel Poverty Action Lab (J-PAL).