NATIONAL BUREAU OF ECONOMIC RESEARCH
NATIONAL BUREAU OF ECONOMIC RESEARCH

How Much Should We Trust Differences-in-Differences Estimates?

Marianne Bertrand, Esther Duflo, Sendhil Mullainathan

NBER Working Paper No. 8841
Issued in March 2002
NBER Program(s):   CH   LS   PE

Most Difference-in-Difference (DD) papers rely on many years of data and focus on serially correlated outcomes. Yet almost all these papers ignore the bias in the estimated standard errors that serial correlation introduce4s. This is especially troubling because the independent variable of interest in DD estimation (e.g., the passage of law) is itself very serially correlated, which will exacerbate the bias in standard errors. To illustrate the severity of this issue, we randomly generate placebo laws in state-level data on female wages from the Current Population Survey. For each law, we use OLS to compute the DD estimate of its 'effect' as well as the standard error for this estimate. The standard errors are severely biased: with about 20 years of data, DD estimation finds an 'effect' significant at the 5% level of up to 45% of the placebo laws. Two very simple techniques can solve this problem for large sample sizes. The first technique consists in collapsing the data and ignoring the time-series variation altogether; the second technique is to estimate standard errors while allowing for an arbitrary covariance structure between time periods. We also suggest a third technique, based on randomization inference testing methods, which works well irrespective of sample size. This technique uses the empirical distribution of estimated effects for placebo laws to form the test distribution.

download in pdf format
   (459 K)

email paper

This paper is available as PDF (459 K) or via email.

Machine-readable bibliographic record - MARC, RIS, BibTeX

Document Object Identifier (DOI): 10.3386/w8841

Published: Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-in-Differences Estimates?," The Quarterly Journal of Economics, MIT Press, vol. 119(1), pages 249-275, February.

Users who downloaded this paper also downloaded these:
Abadie, Diamond, and Hainmueller w12831 Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California's Tobacco Control Program
Lee and Lemieux w14723 Regression Discontinuity Designs in Economics
Duflo w7860 Schooling and Labor Market Consequences of School Construction in Indonesia: Evidence from an Unusual Policy Experiment
Conley and Taber t0312 Inference with "Difference in Differences" with a Small Number of Policy Changes
Card and Krueger w4509 Minimum Wages and Employment: A Case Study of the Fast Food Industry in New Jersey and Pennsylvania
 
Publications
Activities
Meetings
Data
People
About

Support
National Bureau of Economic Research, 1050 Massachusetts Ave., Cambridge, MA 02138; 617-868-3900; email: info@nber.org

Contact Us