- Turkomatic validates the quality of work produced by subdivide, solve, and merge functions by asking the crowd to vote on the best of several redundant answers.
- In verification HITs, a worker is presented with a task and its candidate decompositions or solutions; the worker is asked to vote on the answer that best accomplishes the task.
- Using workers to select from redundant answers for quality control is a common strategy in many crowdsourcing applications.

Boston College

- PDS is a general-purpose algorithm for inducing the crowd to generate workflows for tasks specified by the requester.
- PDS guides workers through the process of converting large, complex tasks into microtasks appropriate for crowd markets. 
- Once all the microtasks for a given component of the workflow are solved, a merge stage asks workers to recombine work back into a coherent single solution to the original task.
- Both stages rely on verification functions that determine when work needs to be checked or repeated.
- Turkomatic posts human intelligence tasks (HIT) on Amazon Mechanical Turk, though the algorithm is independent of this particular platform.

Additional Sections:
- Price-Divide-Solve
- Merge Phase
- Verification

Crowdsourcing Workflows: Algorithmic Model: Price-Divide-Solve, then Merge

Kulkarni, A., Can, M., & Harmann, B. (2012). Collaboratively crowdsourcing workflows with Turkomatic. Association for Computing Machinery, 1003-1012. DOI: https://doi.org/10.1145/2145204.2145354 

Collaboratively crowdsourcing workflows with Turkomatic

- The PDS phase splits work into fairly priced sub-tasks; an initial pricing HIT presents the overall task goal and asks whether or not this task can be solved for a fixed price.
- If a worker indicates the task can be solved directly at the stated price, a new HIT will be posted, asking another worker to do so.
- If the task if judged too complex, the next worker is asked to divide the task into two or more subtasks that are easier to solve than the original task.
- A set of subtasks is then posted to Mechanical Turk.
- This process is recursive; subtasks generated by the subdivide step may themselves be broken down by another step.


Crowdsourcing Workflows: Price-Divide-Solve

- The merge phase combines subtask solutions elements produced during the divide steps. 
- Once all the subtasks produced in a given subdivide step have been solved, the solutions are listed together in a merge HIT, which instructs a worker to combine the subtask solutions in a way that solves the overall goal at the given level of the task hierarchy..
- The process continues until the requester’s original task is solved.

Learn Before

Related