1Cademy - Crowdsourcing Workflows: Standard Quality Assurance Techniques are Insufficient

Concept

Crowdsourcing Workflows: Standard Quality Assurance Techniques are Insufficient

Adding redundancy by asking more workers to contribute and vote on subdivisions at each step failed to yield more successful decomposition.
One explanation for this result is that Turkomatic’s model for redundancy does not allow workers to iterate towards correct answers, but simple increases the parallelism in each step.
Increasingly, eager workers sometimes arrived in already derailed workflows and made efforts to correct the state of the system.
In several cases, workers accepted Turkomatic HITs, but were not satisfied with the workflow provided by previous members of the crowd, and these workers chose to email the authors directly to suggest improvements to the task.
This result suggests that making Turkomatic requester interface available to motivated workers to allow the crowd to self-police and improve its own work would be more effective.

0

1

Updated 2021-07-05

Contributors are:

Who are from:

References