Site Selection Bias in Program Evaluation

Hunt Allcott, Sendhil Mullainathan

NBER Working Paper No. 18373
Issued in September 2012
NBER Program(s):   EEE   LS

“Site selection bias” occurs when the probability that partners adopt or evaluate a program is correlated with treatment effects. I test for site selection bias in the context of the Opower energy conservation programs, using 111 randomized control trials (RCTs) involving 8.6 million households across the United States. Predictions based on rich microdata from the first ten replications substantially overstate efficacy in the next 101 sites. There is evidence of two positive selection mechanisms. First, local populations with stronger preferences for environmental conservation both encourage utilities to adopt the program and are more responsive to the treatment. Second, program managers initially target treatment at the most responsive consumer sub-populations, meaning that efficacy drops when utilities expand the program. While it may be optimal to initially target an intervention toward the most responsive populations, these results show how analysts can be systematically biased when extrapolating experimental results, even after many replications. I augment the Opower results by showing that microfinance institutions (MFIs) that run RCTs differ from the global population of MFIs and that hospitals that host clinical trials differ from the national population of hospitals.

You may purchase this paper on-line in .pdf format from ($5) for electronic delivery.

Information about Free Papers

You should expect a free download if you are a subscriber, a corporate associate of the NBER, a journalist, an employee of the U.S. federal government with a ".GOV" domain name, or a resident of nearly any developing country or transition economy.

If you usually get free papers at work/university but do not at home, you can either connect to your work VPN or proxy (if any) or elect to have a link to the paper emailed to your work email address below. The email address must be connected to a subscribing college, university, or other subscribing institution. Gmail and other free email addresses will not have access.


This paper was revised on March 21, 2014


Machine-readable bibliographic record - MARC, RIS, BibTeX

Document Object Identifier (DOI): 10.3386/w18373

Users who downloaded this paper also downloaded these:
Barro and McCleary w9682 Religion and Economic Growth
Duflo and Hanna w11880 Monitoring Works: Getting Teachers to Come to School
Brueckner and Neumark w16797 Beaches, Sunshine, and Public-Sector Pay: Theory and Evidence on Amenities and Rent Extraction by Government Workers
Levitt, List, Neckermann, and Sadoff w18165 The Behavioralist Goes to School: Leveraging Behavioral Economics to Improve Educational Performance
Gruber w11377 Religious Market Structure, Religious Participation, and Outcomes: Is Religion Good for You?

National Bureau of Economic Research, 1050 Massachusetts Ave., Cambridge, MA 02138; 617-868-3900; email:

Contact Us