- Researchers ask the following question: can the crowd be guided algorithmically to plan and solve problems without any input from requesters?
- Undirected crowd planning seems unlikely to produce usable workflows; design problems often require extensive communication and coordination between the requester and worker, and correct interpretation of a prompt may rely on domain specific knowledge.
- To provide a baseline of success and failure patterns, researchers ran the PDS algorithm with no monitoring by the requester.

Crowdsourcing Workflows: Unsupervised Crowd Planning

- Via Turkomatic, researchers requester that the crowd plan workflows for and compute answers to several types of objectives, including essay writing, natural language queries, itinerary planning, Java programming, and multimedia content generation.
- Researchers ran more than 20 distinct kinds of queries in total, where each HIT earned the worker a different price.
- Workers were given 20 minutes to complete each HIT.

Requests given to the crowd were phrased as follows:
- Essay writing: “write a 3-paragraph essay about crowdsourcing.”
- Natural language query: “create a list of the names of Department Chairs of the top 20 computer science college programs in the US.”
- Itinerary planning: “plan and complete a road trip from San Francisco, California, to New York City. Completely include the locations of all necessary hotels, restaurants, and sights along the way.”
- Java programming: “Please write a short piece of Java code to reverse a string. The algorithm should take as input a string and output its reverse. Make sure it compiles.”

Crowdsourcing Workflows: Procedure

Researchers observed both successes and systematic failures.

Additional Sections:
- Table 1: Experimental Results
- Snap Judgements
- Successful Planning
- Challenges: Task Derailment, Emergent Complexity, and Cycling
- Task Starvation
- Standard Quality Techniques are Insufficient

Crowdsourcing Workflows: Results and Observations

To explore how effectively crowds can be used to support the execution of complex work, researchers performed two evaluations:
- First, researchers examined how crowds performed in producing and solving workflows without expert intervention.
- Second, researchers looked at how expert intervention can improve the crowd’s performance.

Additional Sections:
- Unsupervised Crowd Planning
- Procedure
- Results and Observations

Boston College

The process of preparing jobs for  the crowdsourcing marketplace requires planning, as well as the development of an efficient workflow design. Researchers of this study introduce the program Turkomatic, which aims to recruit crowdworkers to develop workflows for larger tasks, where their goal is to break a job down into microtasks which can then be completed by other crowdsource workers. Results suggest that allowing independent crowdworkers to develop the workflow design is partially successful without requester supervision, but requester supervision during workflow planning and execution improves results dramatically. Researchers argue that Turkomatic’s design will be more successful than the conventional workflow design process.

Collaboratively crowdsourcing workflows with Turkomatic

Kulkarni, A., Can, M., & Harmann, B. (2012). Collaboratively crowdsourcing workflows with Turkomatic. Association for Computing Machinery, 1003-1012. DOI: https://doi.org/10.1145/2145204.2145354 

- Crowdsourcing marketplaces are increasingly used to solve computational problems that require human intelligence (i.e. audio transcription, data verification).
 -Many of these problems are solved via a series of microtasks; multiple microtasks are then chained together into workflows.
 -Such workflows may decompose larger tasks into smaller subtasks, and later recompose subtask solutions into an overall work product.
 -A fundamental challenge in the use of crowdsourcing markets is the workflow design problem: how cam employees divide a complex task into a set of microtasks that can be accurately solved by a pool of crowd workers?
- Effective workflow design remains a major challenge, requiring substantial planning, software development, and testing, which can limit participation in crowdsourcing marketplaces and the kind of work that can be crowdsourced.
- Researchers propose that the responsibility for workflow design can be shared between the crowd and the requester.
- Turkomatic uses a novel meta-workflow to induce the crowd to design and execute workflows, utilizing a continuous price-divide-solve loop that asks workers to recursively divide complex steps into smaller ones until they are at an appropriately simple level, then to solve them.

Crowdsourcing Workflows: Introduction

- Early work in human computation emphasized its utility as a tool for efficiently processing larger datasets in applications like tagging and classification that were outside the reach of autonomous algorithms.
- Most recent research has attempted to expand the types of tasks that can be solved via distributed human computation.
- While crowdsourcing tasks are designed manually based on prior experience and intuition, formal techniques can also play a useful part.
- Both text-based and visual workflow languages have been devised in prior work; some workflow languages have recently been extended to connect to crowdsourcing marketplaces.

Crowdsourcing Workflows: Related Work

Turkomatic consists of two major components:
- First, a meta-workflow uses the crowd to assist in the design of the workflow, and to execute it; the price-divide-solve algorithm (PDS) uses a divide-and-conquer strategy to let workers decide how to split up tasks.
- Second, an editable workflow visualization enables requesters to observe and manage work being executed by the crowd.

Additional Sections:
- Algorithmic Model: Price-Divide-Solve, then Merge
- Task Design Strategies
- Requester Interfaces

Crowdsourcing Workflows: The Turkomatic System

Crowdsourcing Workflows: How Well Can Crowds Plan and Execute?

- The results of the previous section suggest that the crowd can sometimes produce acceptable results in planning and executing solutions to general requests, but more frequently face challenges.
- This section elaborates more on informal experiments for both approaches, first with an expert crowd of workers recruited at UC Berkeley, and the second with a Mechanical Turk crowd.
- Researchers re-ran the essay-writing, blog creation, and natural-language query tasks discussed earlier.

At least two different strategies to address these challenges are plausible:
- The first strategy is to recruit more expert workers.
- The second strategy is to allow requesters to monitor and selectively intervene in the process.

Additional Sections:
- Outcome
- Discussion: Why Does Collaborative Workflow Creation Succeed?

Crowdsourcing Workflows: Collaborative Planning and Execution

- The authors have been using Turkomatic as a platform since its creation to test a variety of tasks, and have also shared the program with a number of end users within academia and the computer science community.
- Outcomes illustrate two additional benefits of Turkomatic: it has value in quickly evaluating the ability of crowds to solve particular kinds of work, and can reduce the complexity of accessing crowd platforms for causal use.
- Researchers hypothesize that crowdsourcing marketplaces can grow substantially if the process of sending work to crowds is simplified.

Crowdsourcing Workflows: User Experiences with Turkomatic

Researchers list several observations that arise from their work with the Turkomatic system on collaboratively designing workflows with the crowd.

Additional Sections:
- Instructional Writing is Difficult for Workers
- Reputation Effects Arise from Worker Planning
- Excessive Structure Limits and Effectiveness of Leaders
- Scaling Requires Context-Free Workflows

Crowdsourcing Workflows: Discussion

- This paper introduced Turkomatic, a system that harnesses crowds to design and execute workflows for complex tasks. 
- The system is based on a price-divide-price algorithm that guides workers through the steps of decomposing and solving tasks.
- Researchers’ experience with Turkomatic suggests that unsupervised workers face planning challenges, but experiments also showed that planning succeeds for an interesting range of tasks when more knowledgeable workers are recruited and when requesters can review and edit crowd work in real-time.
- The one-size-fits-all model of Turkomatic trades off simplicity of use for runtime supervision; workflows can be generated without exhaustive planning, but require requester monitoring at runtime to guarantee quality of results.

Learn Before

Related

Learn After