Testing the Waters: Behavior across Participant Pools
We leverage a large-scale incentivized survey eliciting behaviors from (almost) an entire university student population, a representative sample of the U.S. population, and Amazon Mechanical Turk (MTurk) to address concerns about the external validity of experiments with student participants. Behavior in the student population offers bounds on behaviors in other populations, and correlations between behaviors are largely similar across samples. Furthermore, non-student samples exhibit higher measurement error. Adding historical lab participation data, we find a small set of attributes over which lab participants differ from non-lab participants. Using an additional set of lab experiments, we see no evidence of observer effects.
Snowberg gratefully acknowledges the support of NSF grants SES-1156154 and SMA-1329195. Yariv gratefully acknowledges the support of NSF grant SES-1629613 and the Gordon and Betty Moore Foundation grant 1158. We thank Marina Agranov, Alessandra Casella, Armin Falk, Guillaume Frechette, Drew Fudenberg, Johannes Haushoffer, Salvatore Nunnari, Nichole Szembrot, Emanuel Vespa, and seminar audiences at Columbia University, Cornell, NTU, UCSD, and University of Maryland for useful comments and encouragement. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.
Erik Snowberg & Leeat Yariv, 2021. "Testing the Waters: Behavior across Participant Pools," American Economic Review, vol 111(2), pages 687-719.