According to this article, “about 10% of workers are responsible for completing 41% of tasks” on Mturk. Higher rated participants are more likely to qualify for studies that are more complex, even though they may not qualify for these studies. Because workers are more likely to choose tasks they are more familiar with, we may not gain a truly random sample for our study.

DePaul University

1.  Mturk workers are not a representative sample
2. Workers may be more willing to cheat
3. Practice effects
4. Less likely to get a random sample

Cons for using MTurk as participant pool

Paolacci, G., & Chandler, J. (2014). Current Directions in Psychological Science, 23(3) 184-188. https://doi.org/10.1177/0963721414531598. 

Inside the Turk: Understanding 
Mechanical Turk as a Participant Pool

Mturk can be used as a supplement for choosing participants but the workers are not a representative sample of the population as a whole, because of location, demographics, and personality. Most Mtukers are in the United States and India, are usually around 30 years old, are overeducated, underemployed, and liberal. They are also more introverted and neurotic than college aged participants. 

Not a representative sample

Mturkers may be more likely to cheat on evaluations compared to college students, due to lack of monitoring and social desireability. As of this paper, there was not a way to remotely monitor participants as they completed tasks, on Mturk. This may give workers more incentive to look up answers on the Internet. Likewise, workers are more likely to exhibit social desireability personalities than college students. Workers want to both please the researcher and qualify for better paid studies. This means they are more likely to choose responses they think will please the researcher, rather than their true opinions. 

More likely to cheat

May not get a random sample

A practice effect is a specific type of carryover effect in within-subjects experiments where participants perform a task better in later experimental conditions. This improvement occurs not because of the experimental manipulation, but simply because the participants have gained experience and had the opportunity to practice the task during earlier conditions.

Learn Before

Related