Are experiments that recruit from online subject pools field experiments or laboratory experiments?
Recently, I attended a virtual seminar where one of my colleagues presented an experiment that recruited participants from an online labor market like Amazon Mechanical Turk and Prolific. The experiment elicited dependent variables, manipulated independent variables, and randomly allocated participants to each treatment. This colleague referred to the experiment as a “field experiment,” which provoked quite some criticism from the audience members. Observing this unfold, I wondered whether it is appropriate to classify this type of experiment as a field experiment. However ridiculous this may sound to some, the alternative classification, i.e., laboratory experiment, does not seem to fit well either.
If it were up to John List, an authority on field experiments, experiments that recruit from online labor markets would be classified as field experiments (Floyd and List, 2016; Harrison and List, 2004). Specifically, his taxonomy would classify such experiments as “artefactual field experiments” or “framed field experiments” because they use a non-student subject pool. We could follow this taxonomy, but careful analysis reveals that it classifies nearly any experiment that does not feature a student subject pool as a field experiment. This is unsatisfactory because the border between student and non-student samples are blurred in many of accounting's topical areas. Are audit professionals pursuing an audit qualification students or practitioners? What about professionals enrolled in a parttime MBA program?
The accounting literature also provides some guidance on classifying online experiments (e.g., Bonner 2007, Libby et al. 2002, Bloomfield et al. 2016). For instance, Bloomfield et al. (2016) would classify such experiments as laboratory experiments: They explain in footnote 19 that the term “laboratory” in laboratory experiment should not be taken literally. Instead, it refers to the experiment eliciting dependent variables in a self-constructed setting.1
We could follow Bloomfield et al. (2016) in referring to experiments that recruit online participants as laboratory experiments. However, conducting experiments in a physical lab with students is quite different from conducting them with MTurk of Prolific participants in an online environment. Student samples tend to be “WEIRD”—Western, Educated, Industrialized, Rich, and Democratic (Henrich et al., 2010), while samples drawn from online labor markets are more heterogenous. Also, physical labs provide more control over the setting in which dependent variables are elicited than online environments.
Perhaps it is time to give experiments that recruit participants from online labor markets their own category. Can’t we just refer to them as “online experiments”? Giving these experiments their own category would also enable us to have a more informed, long-term discussion about their benefits and costs compared to the other types of experiments. It would also help us discuss important topics like population of interest. Some types of experiments and samples may be better suited when targetting some populations over others.
References
How to reference this online article?
Bloomfield et al. (2016) also explain that laboratory experiments differ from field experiments because participants are unaware they are being studied when taking part in a field experiment. A field experiment they define as a study that observes dependent variables in naturally-occuring settings while manipulating an independent variable. This definition of field experiments shows similarities to the most conservative category in List’s taxonomy: natural field experiments.↩
