- Turkomatic assigns responsibility for working tasks to the workers, which can mean poor choices by the Turkers may negatively affect the requester’s reputation.
- Researchers noted that posts on a Turk discussion board complained of poor working or excessive scope on some requests, and these workers seemed unaware that these tasks were created by other workers.

Boston College

Researchers list several observations that arise from their work with the Turkomatic system on collaboratively designing workflows with the crowd.

Additional Sections:
- Instructional Writing is Difficult for Workers
- Reputation Effects Arise from Worker Planning
- Excessive Structure Limits and Effectiveness of Leaders
- Scaling Requires Context-Free Workflows

Crowdsourcing Workflows: Discussion

Kulkarni, A., Can, M., & Harmann, B. (2012). Collaboratively crowdsourcing workflows with Turkomatic. Association for Computing Machinery, 1003-1012. DOI: https://doi.org/10.1145/2145204.2145354 

Collaboratively crowdsourcing workflows with Turkomatic

- Composing good instructions is not trivial and takes time and effort.
- This particular requirement stands in tension with the goal of workers to maximize the number of tasks they complete per unit of time.
- While some workers were able to write detailed instructions, few other tasks on Mechanical Turk require such attention, and workers may thus be incentivized from delivering quality work.

Crowdsourcing Workflows: Instructional Writing is Difficult for Workers

Crowdsourcing Workflows: Reputation Effects Arise from Worker Planning

- Researchers noted that expert workers provided more detailed instructions in their subdivisions and attempted to communicate corrections to requesters through external channels of communication.
- The PDS algorithm does not yet offer workers the ability to contribute at different levels commensurate with skill and interest.
- This phenomenon suggests that more effective platforms for collaboration with the crowd should permit workers to edit workflows much as requesters do in the current system.

Crowdsourcing Workflows: Excessive Structure Limits and Effectiveness of Leaders

- Batch processes send many different problem instances through the same workflow, this implies that such workflows must be designed in a context-free manner.
- Turkomatic currently does not guarantee that crowd-planned workflows are context-free, but a promising area for future work is to first ask the crowd to produce a concrete workflow, and after generalize it to fit multiple problem instances.

Learn Before

Related