Paolacci, G., & Chandler, J. (2014). Current Directions in Psychological Science, 23(3) 184-188. https://doi.org/10.1177/0963721414531598. 

Inside the Turk: Understanding 
Mechanical Turk as a Participant Pool

Mechanical Turk (MTurk) is an Amazon created online job marketplace built for internet workers. The marketplace is created for companies to provide tasks for Mturkers. Mturkers are paid to complete low skill tasks, such as data entry or data cleaning.

Mechanical Turk

There are two prospective on the impact of rewarding on participants' performances.
- Economic behaviorists
- SDT self determination theory
- Using Mturk as a participant pool 


University of Michigan - Ann Arbor

DePaul University

- Providing participants with no incentive or reward
- Taking per-test MCQ
- Providing Sample passage before reading actual map
- Free Recall Scoring
- Qualitative Coding of feedback

Experiment set-up for Hybrid map paper

Rewarding participants Good or bad?

In our experiment before displaying the actual maps to be studied, we presented a sample map with different content and each researcher verbally explained the structure and trained students how to navigate and read the map. Participants were asked to navigate through the maps and try different features until they feel comfortable with the format and structure then they were allowed to proceed. This process is aimed to reduce the “Map Shock” effect on participants as discussed by[dansereau2005node].

Providing Sample passage before reading actual map

1. Scoring of the free-recall responses were based on a procedure developed
by (Meyer (1975)) and (Holley, Dansereau, McDonald, Garland, and Collins
(1979) Pearson correlation coefficient of 0.86) as explained by [Hall 1996].
2. We divided the original material into an inclusive set of propositions. Each proposition contained one fact. Four experienced scorers matched these
propositions with the set of propositions contained in a participant’s free-recall
test [Hall 1996]. 
3. Reliability was established by having 4 experienced rater
score a randomly selected subset of each recall response. A minimum reliability
coefficient of 75% was guaranteed.


Scoring of free recall responses

- Hall, R. H., & O'Donnell, A. (1996). Cognitive and affective outcomes of learning from knowledge maps. Contemporary educational psychology, 21(1), 94-101.
- Holley, C. D., Dansereau, D. F., McDonald, B. A., Garland, J. C., & Collins, K. W. (1979). Evaluation of a hierarchical mapping technique as an aid to prose processing. Contemporary Educational Psychology, 4(3), 227-237.
- Meyer, B. J. (1975). Identification of the structure of prose and its implications for the study of reading and memory. Journal of Reading Behavior, 7(1), 7-47.

Learn Before

Related

Learn After