- PDS is a general-purpose algorithm for inducing the crowd to generate workflows for tasks specified by the requester.
- PDS guides workers through the process of converting large, complex tasks into microtasks appropriate for crowd markets. 
- Once all the microtasks for a given component of the workflow are solved, a merge stage asks workers to recombine work back into a coherent single solution to the original task.
- Both stages rely on verification functions that determine when work needs to be checked or repeated.
- Turkomatic posts human intelligence tasks (HIT) on Amazon Mechanical Turk, though the algorithm is independent of this particular platform.

Additional Sections:
- Price-Divide-Solve
- Merge Phase
- Verification

Crowdsourcing Workflows: Algorithmic Model: Price-Divide-Solve, then Merge

- Designing general-purpose HITs that fit a wide variety of tasks while still conveying specific requirements of the decomposition and merge phases to a worker is challenging.
- Two design techniques proceed especially useful and necessary: including larger task context in subtasks, and visually separating task language from context language.

Additional Sections:
- Show Context of Tasks in a Workflow
- Visually Separate Task Language from Context Language

Crowdsourcing Workflows: Task Design Strategies

This section reviews interaction with Turkomatic from the requester’s perspective.

Additional Sections:
- Requesting Work
- Visualizing Ongoing Work
- Editing Workflows

Crowdsourcing Workflows: Requester Interfaces

Turkomatic consists of two major components:
- First, a meta-workflow uses the crowd to assist in the design of the workflow, and to execute it; the price-divide-solve algorithm (PDS) uses a divide-and-conquer strategy to let workers decide how to split up tasks.
- Second, an editable workflow visualization enables requesters to observe and manage work being executed by the crowd.

Additional Sections:
- Algorithmic Model: Price-Divide-Solve, then Merge
- Task Design Strategies
- Requester Interfaces

Boston College

The process of preparing jobs for  the crowdsourcing marketplace requires planning, as well as the development of an efficient workflow design. Researchers of this study introduce the program Turkomatic, which aims to recruit crowdworkers to develop workflows for larger tasks, where their goal is to break a job down into microtasks which can then be completed by other crowdsource workers. Results suggest that allowing independent crowdworkers to develop the workflow design is partially successful without requester supervision, but requester supervision during workflow planning and execution improves results dramatically. Researchers argue that Turkomatic’s design will be more successful than the conventional workflow design process.

Collaboratively crowdsourcing workflows with Turkomatic

Kulkarni, A., Can, M., & Harmann, B. (2012). Collaboratively crowdsourcing workflows with Turkomatic. Association for Computing Machinery, 1003-1012. DOI: https://doi.org/10.1145/2145204.2145354 

- Crowdsourcing marketplaces are increasingly used to solve computational problems that require human intelligence (i.e. audio transcription, data verification).
 -Many of these problems are solved via a series of microtasks; multiple microtasks are then chained together into workflows.
 -Such workflows may decompose larger tasks into smaller subtasks, and later recompose subtask solutions into an overall work product.
 -A fundamental challenge in the use of crowdsourcing markets is the workflow design problem: how cam employees divide a complex task into a set of microtasks that can be accurately solved by a pool of crowd workers?
- Effective workflow design remains a major challenge, requiring substantial planning, software development, and testing, which can limit participation in crowdsourcing marketplaces and the kind of work that can be crowdsourced.
- Researchers propose that the responsibility for workflow design can be shared between the crowd and the requester.
- Turkomatic uses a novel meta-workflow to induce the crowd to design and execute workflows, utilizing a continuous price-divide-solve loop that asks workers to recursively divide complex steps into smaller ones until they are at an appropriately simple level, then to solve them.

Crowdsourcing Workflows: Introduction

- Early work in human computation emphasized its utility as a tool for efficiently processing larger datasets in applications like tagging and classification that were outside the reach of autonomous algorithms.
- Most recent research has attempted to expand the types of tasks that can be solved via distributed human computation.
- While crowdsourcing tasks are designed manually based on prior experience and intuition, formal techniques can also play a useful part.
- Both text-based and visual workflow languages have been devised in prior work; some workflow languages have recently been extended to connect to crowdsourcing marketplaces.

Crowdsourcing Workflows: Related Work

Crowdsourcing Workflows: The Turkomatic System

To explore how effectively crowds can be used to support the execution of complex work, researchers performed two evaluations:
- First, researchers examined how crowds performed in producing and solving workflows without expert intervention.
- Second, researchers looked at how expert intervention can improve the crowd’s performance.

Additional Sections:
- Unsupervised Crowd Planning
- Procedure
- Results and Observations

Crowdsourcing Workflows: How Well Can Crowds Plan and Execute?

- The results of the previous section suggest that the crowd can sometimes produce acceptable results in planning and executing solutions to general requests, but more frequently face challenges.
- This section elaborates more on informal experiments for both approaches, first with an expert crowd of workers recruited at UC Berkeley, and the second with a Mechanical Turk crowd.
- Researchers re-ran the essay-writing, blog creation, and natural-language query tasks discussed earlier.

At least two different strategies to address these challenges are plausible:
- The first strategy is to recruit more expert workers.
- The second strategy is to allow requesters to monitor and selectively intervene in the process.

Additional Sections:
- Outcome
- Discussion: Why Does Collaborative Workflow Creation Succeed?

Crowdsourcing Workflows: Collaborative Planning and Execution

- The authors have been using Turkomatic as a platform since its creation to test a variety of tasks, and have also shared the program with a number of end users within academia and the computer science community.
- Outcomes illustrate two additional benefits of Turkomatic: it has value in quickly evaluating the ability of crowds to solve particular kinds of work, and can reduce the complexity of accessing crowd platforms for causal use.
- Researchers hypothesize that crowdsourcing marketplaces can grow substantially if the process of sending work to crowds is simplified.

Crowdsourcing Workflows: User Experiences with Turkomatic

Researchers list several observations that arise from their work with the Turkomatic system on collaboratively designing workflows with the crowd.

Additional Sections:
- Instructional Writing is Difficult for Workers
- Reputation Effects Arise from Worker Planning
- Excessive Structure Limits and Effectiveness of Leaders
- Scaling Requires Context-Free Workflows

Crowdsourcing Workflows: Discussion

- This paper introduced Turkomatic, a system that harnesses crowds to design and execute workflows for complex tasks. 
- The system is based on a price-divide-price algorithm that guides workers through the steps of decomposing and solving tasks.
- Researchers’ experience with Turkomatic suggests that unsupervised workers face planning challenges, but experiments also showed that planning succeeds for an interesting range of tasks when more knowledgeable workers are recruited and when requesters can review and edit crowd work in real-time.
- The one-size-fits-all model of Turkomatic trades off simplicity of use for runtime supervision; workflows can be generated without exhaustive planning, but require requester monitoring at runtime to guarantee quality of results.

Learn Before

Related

Learn After