- Composing good instructions is not trivial and takes time and effort.
- This particular requirement stands in tension with the goal of workers to maximize the number of tasks they complete per unit of time.
- While some workers were able to write detailed instructions, few other tasks on Mechanical Turk require such attention, and workers may thus be incentivized from delivering quality work.

Crowdsourcing Workflows: Instructional Writing is Difficult for Workers

- Turkomatic assigns responsibility for working tasks to the workers, which can mean poor choices by the Turkers may negatively affect the requester’s reputation.
- Researchers noted that posts on a Turk discussion board complained of poor working or excessive scope on some requests, and these workers seemed unaware that these tasks were created by other workers.

Crowdsourcing Workflows: Reputation Effects Arise from Worker Planning

- Researchers noted that expert workers provided more detailed instructions in their subdivisions and attempted to communicate corrections to requesters through external channels of communication.
- The PDS algorithm does not yet offer workers the ability to contribute at different levels commensurate with skill and interest.
- This phenomenon suggests that more effective platforms for collaboration with the crowd should permit workers to edit workflows much as requesters do in the current system.

Crowdsourcing Workflows: Excessive Structure Limits and Effectiveness of Leaders

- Batch processes send many different problem instances through the same workflow, this implies that such workflows must be designed in a context-free manner.
- Turkomatic currently does not guarantee that crowd-planned workflows are context-free, but a promising area for future work is to first ask the crowd to produce a concrete workflow, and after generalize it to fit multiple problem instances.

Crowdsourcing Workflows: Scaling Requires Context-Free Workflows

Researchers list several observations that arise from their work with the Turkomatic system on collaboratively designing workflows with the crowd.

Additional Sections:
- Instructional Writing is Difficult for Workers
- Reputation Effects Arise from Worker Planning
- Excessive Structure Limits and Effectiveness of Leaders
- Scaling Requires Context-Free Workflows

Boston College

The process of preparing jobs for  the crowdsourcing marketplace requires planning, as well as the development of an efficient workflow design. Researchers of this study introduce the program Turkomatic, which aims to recruit crowdworkers to develop workflows for larger tasks, where their goal is to break a job down into microtasks which can then be completed by other crowdsource workers. Results suggest that allowing independent crowdworkers to develop the workflow design is partially successful without requester supervision, but requester supervision during workflow planning and execution improves results dramatically. Researchers argue that Turkomatic’s design will be more successful than the conventional workflow design process.

Collaboratively crowdsourcing workflows with Turkomatic

Kulkarni, A., Can, M., & Harmann, B. (2012). Collaboratively crowdsourcing workflows with Turkomatic. Association for Computing Machinery, 1003-1012. DOI: https://doi.org/10.1145/2145204.2145354 

- Crowdsourcing marketplaces are increasingly used to solve computational problems that require human intelligence (i.e. audio transcription, data verification).
 -Many of these problems are solved via a series of microtasks; multiple microtasks are then chained together into workflows.
 -Such workflows may decompose larger tasks into smaller subtasks, and later recompose subtask solutions into an overall work product.
 -A fundamental challenge in the use of crowdsourcing markets is the workflow design problem: how cam employees divide a complex task into a set of microtasks that can be accurately solved by a pool of crowd workers?
- Effective workflow design remains a major challenge, requiring substantial planning, software development, and testing, which can limit participation in crowdsourcing marketplaces and the kind of work that can be crowdsourced.
- Researchers propose that the responsibility for workflow design can be shared between the crowd and the requester.
- Turkomatic uses a novel meta-workflow to induce the crowd to design and execute workflows, utilizing a continuous price-divide-solve loop that asks workers to recursively divide complex steps into smaller ones until they are at an appropriately simple level, then to solve them.

Crowdsourcing Workflows: Introduction

- Early work in human computation emphasized its utility as a tool for efficiently processing larger datasets in applications like tagging and classification that were outside the reach of autonomous algorithms.
- Most recent research has attempted to expand the types of tasks that can be solved via distributed human computation.
- While crowdsourcing tasks are designed manually based on prior experience and intuition, formal techniques can also play a useful part.
- Both text-based and visual workflow languages have been devised in prior work; some workflow languages have recently been extended to connect to crowdsourcing marketplaces.

Crowdsourcing Workflows: Related Work

Turkomatic consists of two major components:
- First, a meta-workflow uses the crowd to assist in the design of the workflow, and to execute it; the price-divide-solve algorithm (PDS) uses a divide-and-conquer strategy to let workers decide how to split up tasks.
- Second, an editable workflow visualization enables requesters to observe and manage work being executed by the crowd.

Additional Sections:
- Algorithmic Model: Price-Divide-Solve, then Merge
- Task Design Strategies
- Requester Interfaces

Crowdsourcing Workflows: The Turkomatic System

To explore how effectively crowds can be used to support the execution of complex work, researchers performed two evaluations:
- First, researchers examined how crowds performed in producing and solving workflows without expert intervention.
- Second, researchers looked at how expert intervention can improve the crowd’s performance.

Additional Sections:
- Unsupervised Crowd Planning
- Procedure
- Results and Observations

Crowdsourcing Workflows: How Well Can Crowds Plan and Execute?

- The results of the previous section suggest that the crowd can sometimes produce acceptable results in planning and executing solutions to general requests, but more frequently face challenges.
- This section elaborates more on informal experiments for both approaches, first with an expert crowd of workers recruited at UC Berkeley, and the second with a Mechanical Turk crowd.
- Researchers re-ran the essay-writing, blog creation, and natural-language query tasks discussed earlier.

At least two different strategies to address these challenges are plausible:
- The first strategy is to recruit more expert workers.
- The second strategy is to allow requesters to monitor and selectively intervene in the process.

Additional Sections:
- Outcome
- Discussion: Why Does Collaborative Workflow Creation Succeed?

Crowdsourcing Workflows: Collaborative Planning and Execution

- The authors have been using Turkomatic as a platform since its creation to test a variety of tasks, and have also shared the program with a number of end users within academia and the computer science community.
- Outcomes illustrate two additional benefits of Turkomatic: it has value in quickly evaluating the ability of crowds to solve particular kinds of work, and can reduce the complexity of accessing crowd platforms for causal use.
- Researchers hypothesize that crowdsourcing marketplaces can grow substantially if the process of sending work to crowds is simplified.

Crowdsourcing Workflows: User Experiences with Turkomatic

Crowdsourcing Workflows: Discussion

- This paper introduced Turkomatic, a system that harnesses crowds to design and execute workflows for complex tasks. 
- The system is based on a price-divide-price algorithm that guides workers through the steps of decomposing and solving tasks.
- Researchers’ experience with Turkomatic suggests that unsupervised workers face planning challenges, but experiments also showed that planning succeeds for an interesting range of tasks when more knowledgeable workers are recruited and when requesters can review and edit crowd work in real-time.
- The one-size-fits-all model of Turkomatic trades off simplicity of use for runtime supervision; workflows can be generated without exhaustive planning, but require requester monitoring at runtime to guarantee quality of results.

Learn Before

Related

Learn After