Mturk can be used as a supplement for choosing participants but the workers are not a representative sample of the population as a whole, because of location, demographics, and personality. Most Mtukers are in the United States and India, are usually around 30 years old, are overeducated, underemployed, and liberal. They are also more introverted and neurotic than college aged participants. 

Not a representative sample

Mturkers may be more likely to cheat on evaluations compared to college students, due to lack of monitoring and social desireability. As of this paper, there was not a way to remotely monitor participants as they completed tasks, on Mturk. This may give workers more incentive to look up answers on the Internet. Likewise, workers are more likely to exhibit social desireability personalities than college students. Workers want to both please the researcher and qualify for better paid studies. This means they are more likely to choose responses they think will please the researcher, rather than their true opinions. 

More likely to cheat

According to this article, “about 10% of workers are responsible for completing 41% of tasks” on Mturk. Higher rated participants are more likely to qualify for studies that are more complex, even though they may not qualify for these studies. Because workers are more likely to choose tasks they are more familiar with, we may not gain a truly random sample for our study.

May not get a random sample

A practice effect is a specific type of carryover effect in within-subjects experiments where participants perform a task better in later experimental conditions. This improvement occurs not because of the experimental manipulation, but simply because the participants have gained experience and had the opportunity to practice the task during earlier conditions.

Practice Effect

1.  Mturk workers are not a representative sample
2. Workers may be more willing to cheat
3. Practice effects
4. Less likely to get a random sample

DePaul University

Mechanical Turk (MTurk) is an Amazon created online job marketplace built for internet workers. The marketplace is created for companies to provide tasks for Mturkers. Mturkers are paid to complete low skill tasks, such as data entry or data cleaning.

Mechanical Turk

Paolacci, G., & Chandler, J. (2014). Current Directions in Psychological Science, 23(3) 184-188. https://doi.org/10.1177/0963721414531598. 

Inside the Turk: Understanding 
Mechanical Turk as a Participant Pool

1. Workers have extrinsic and intrinsic motivation
2. Researchers have more control over participant pool
3. Performance is similar between college students and Mturkers

Pros for using MTurk as participant pool

Cons for using MTurk as participant pool

Mechanical Turk (MTurk) is an online crowdsourcing marketplace created by Amazon that serves as a valuable participant recruitment tool for survey researchers. Originally designed for simple usability testing, MTurk provides access to a massive, diverse global database of workers. Researchers can use it to test questionnaire items, manipulate sampling parameters, and rapidly deploy experiments at a very low cost to gather high-quality data.

Learn Before

Related

Learn After