Boston College

Researchers observed both successes and systematic failures.

Additional Sections:
- Table 1: Experimental Results
- Snap Judgements
- Successful Planning
- Challenges: Task Derailment, Emergent Complexity, and Cycling
- Task Starvation
- Standard Quality Techniques are Insufficient

Crowdsourcing Workflows: Results and Observations

Kulkarni, A., Can, M., & Harmann, B. (2012). Collaboratively crowdsourcing workflows with Turkomatic. Association for Computing Machinery, 1003-1012. DOI: https://doi.org/10.1145/2145204.2145354 

Collaboratively crowdsourcing workflows with Turkomatic

Crowdsourcing Workflows: Table 1: Experimental Results

- One class of successes were snap judgements: cases where there was no decomposition of the input task, but where a worker marked the first task as solvable, and a subsequent worker provided a correct, coherent, and complete answer.
- In these outcomes, a correct result is produced without additional decomposition or planning.
- This result is unsurprising, as no collaboration is involved when no workflow is required: the same worker both determines how to solve a task and solves it.

Crowdsourcing Workflows: Snap Judgements

- In some cases, groups of workers broke down tasks in line with requester intentions, and this was sustained over multiple iterations.
- This outcome occurred infrequently, which is not surprising to researchers; it required a shared cognitive model of how work should be decomposed to be maintained by multiple workers in the execution process.
- Researchers found that active participation of one or more highly eager workers early in the princess made a substantial difference in downstream quality.

Crowdsourcing Workflows: Successful Planning

- In the majority of cases, the unsupervised crowd produced unsuitable workflows, or unsuccessful results.
- The most frequent type of failure was derailment, a phenomenon that occurred when the PDS algorithm failed to terminate and continued to produce steps indefinitely.

Derailment can happen for multiple reasons: 
- First, workers were confused about appropriate task granularity.
- Second, workers authored subdivisions that could be executed by a single worker, but not split across separate workers.
- Finally, workers who had lost the context of the overall workflow generated decompositions that restated previous tasks as subtasks, leading to cyclic behavior.

Crowdsourcing Workflows: Challenges: Task Derailment, Emergent Complexity, and Cycling

- Some workflows failed to complete due to starvation; after a while, no new workers attempted the available tasks, and the time limit for the experiment expired without the execution of work continuing further.
- Task starvation has been observed in other projects, and has been counteracted thought listing optimization or chaining sets of tasks to retain workers.
- In the context of the PDS algorithm, starvation occurred most often when a worker marked a task as solvable when, judging by its complexity, it should have been subdivided.

Crowdsourcing Workflows: Task Starvation

- Adding redundancy by asking more workers to contribute and vote on subdivisions at each step failed to yield more successful decomposition.
- One explanation for this result is that Turkomatic’s model for redundancy does not allow workers to iterate towards correct answers, but simple increases the parallelism in each step.
- Increasingly, eager workers sometimes arrived in already derailed workflows and made efforts to correct the state of the system.
- In several cases, workers accepted Turkomatic HITs, but were not satisfied with the workflow provided by previous members of the crowd, and these workers chose to email the authors directly to suggest improvements to the task.
- This result suggests that making Turkomatic requester interface available to motivated workers to allow the crowd to self-police and improve its own work would be more effective.

Learn Before

Related