Kulkarni, A., Can, M., & Harmann, B. (2012). Collaboratively crowdsourcing workflows with Turkomatic. Association for Computing Machinery, 1003-1012. DOI: https://doi.org/10.1145/2145204.2145354 

Collaboratively crowdsourcing workflows with Turkomatic

- Crowdsourcing marketplaces are increasingly used to solve computational problems that require human intelligence (i.e. audio transcription, data verification).
 -Many of these problems are solved via a series of microtasks; multiple microtasks are then chained together into workflows.
 -Such workflows may decompose larger tasks into smaller subtasks, and later recompose subtask solutions into an overall work product.
 -A fundamental challenge in the use of crowdsourcing markets is the workflow design problem: how cam employees divide a complex task into a set of microtasks that can be accurately solved by a pool of crowd workers?
- Effective workflow design remains a major challenge, requiring substantial planning, software development, and testing, which can limit participation in crowdsourcing marketplaces and the kind of work that can be crowdsourced.
- Researchers propose that the responsibility for workflow design can be shared between the crowd and the requester.
- Turkomatic uses a novel meta-workflow to induce the crowd to design and execute workflows, utilizing a continuous price-divide-solve loop that asks workers to recursively divide complex steps into smaller ones until they are at an appropriately simple level, then to solve them.

Crowdsourcing Workflows: Introduction

- Early work in human computation emphasized its utility as a tool for efficiently processing larger datasets in applications like tagging and classification that were outside the reach of autonomous algorithms.
- Most recent research has attempted to expand the types of tasks that can be solved via distributed human computation.
- While crowdsourcing tasks are designed manually based on prior experience and intuition, formal techniques can also play a useful part.
- Both text-based and visual workflow languages have been devised in prior work; some workflow languages have recently been extended to connect to crowdsourcing marketplaces.

Crowdsourcing Workflows: Related Work

Turkomatic consists of two major components:
- First, a meta-workflow uses the crowd to assist in the design of the workflow, and to execute it; the price-divide-solve algorithm (PDS) uses a divide-and-conquer strategy to let workers decide how to split up tasks.
- Second, an editable workflow visualization enables requesters to observe and manage work being executed by the crowd.

Additional Sections:
- Algorithmic Model: Price-Divide-Solve, then Merge
- Task Design Strategies
- Requester Interfaces

Crowdsourcing Workflows: The Turkomatic System

To explore how effectively crowds can be used to support the execution of complex work, researchers performed two evaluations:
- First, researchers examined how crowds performed in producing and solving workflows without expert intervention.
- Second, researchers looked at how expert intervention can improve the crowd’s performance.

Additional Sections:
- Unsupervised Crowd Planning
- Procedure
- Results and Observations

Crowdsourcing Workflows: How Well Can Crowds Plan and Execute?

- The results of the previous section suggest that the crowd can sometimes produce acceptable results in planning and executing solutions to general requests, but more frequently face challenges.
- This section elaborates more on informal experiments for both approaches, first with an expert crowd of workers recruited at UC Berkeley, and the second with a Mechanical Turk crowd.
- Researchers re-ran the essay-writing, blog creation, and natural-language query tasks discussed earlier.

At least two different strategies to address these challenges are plausible:
- The first strategy is to recruit more expert workers.
- The second strategy is to allow requesters to monitor and selectively intervene in the process.

Additional Sections:
- Outcome
- Discussion: Why Does Collaborative Workflow Creation Succeed?

Crowdsourcing Workflows: Collaborative Planning and Execution

- The authors have been using Turkomatic as a platform since its creation to test a variety of tasks, and have also shared the program with a number of end users within academia and the computer science community.
- Outcomes illustrate two additional benefits of Turkomatic: it has value in quickly evaluating the ability of crowds to solve particular kinds of work, and can reduce the complexity of accessing crowd platforms for causal use.
- Researchers hypothesize that crowdsourcing marketplaces can grow substantially if the process of sending work to crowds is simplified.

Crowdsourcing Workflows: User Experiences with Turkomatic

Researchers list several observations that arise from their work with the Turkomatic system on collaboratively designing workflows with the crowd.

Additional Sections:
- Instructional Writing is Difficult for Workers
- Reputation Effects Arise from Worker Planning
- Excessive Structure Limits and Effectiveness of Leaders
- Scaling Requires Context-Free Workflows

Crowdsourcing Workflows: Discussion

- This paper introduced Turkomatic, a system that harnesses crowds to design and execute workflows for complex tasks. 
- The system is based on a price-divide-price algorithm that guides workers through the steps of decomposing and solving tasks.
- Researchers’ experience with Turkomatic suggests that unsupervised workers face planning challenges, but experiments also showed that planning succeeds for an interesting range of tasks when more knowledgeable workers are recruited and when requesters can review and edit crowd work in real-time.
- The one-size-fits-all model of Turkomatic trades off simplicity of use for runtime supervision; workflows can be generated without exhaustive planning, but require requester monitoring at runtime to guarantee quality of results.

Crowdsourcing Workflows: Conclusion

The process of preparing jobs for  the crowdsourcing marketplace requires planning, as well as the development of an efficient workflow design. Researchers of this study introduce the program Turkomatic, which aims to recruit crowdworkers to develop workflows for larger tasks, where their goal is to break a job down into microtasks which can then be completed by other crowdsource workers. Results suggest that allowing independent crowdworkers to develop the workflow design is partially successful without requester supervision, but requester supervision during workflow planning and execution improves results dramatically. Researchers argue that Turkomatic’s design will be more successful than the conventional workflow design process.

Boston College

A workflow consists of an orchestrated and repeatable pattern of activity, enabled by the systematic organization of resources into processes that transform materials, provide services, or process information. It can be depicted as a sequence of operations, the work of a person or group,the work of an organization of staff, or one or more simple or complex mechanisms. From a more abstract or higher-level perspective, workflow may be considered a view or representation of real work. The flow being described may refer to a document, service, or product that is being transferred from one step to another. Workflows may be viewed as one fundamental building block to be combined with other parts of an organization's structure such as information technology, teams, projects and hierarchies.

Workflows

The practice of completing microtasks by using a group of remote workers.

Microtask Crowdsourcing

Citizen science is research contributed by nonprofessional participants via crowdsourcing. In citizen science projects, participants either collect data with mobile technologies or analyze existing data through internet-based Virtual Citizen Science (VCS) platforms, while in the accompaniment of a professional scientist or researcher.
This research improves the scientific community's capacity and increases the public's understanding of science.

Citizen science

Conversational interfaces can facilitate human-computer interactions, but researchers still speculate as to if conversational interfaces can improve worker experience and work quality in crowdsourcing marketplaces. Researchers explore this concept by designing Chatterbox, aimed at gauging the interest and acceptance by crowdworkers for this type of interface. Results suggest that conversational interfaces can be used effectively for crowdsourcing microtasks, resulting in high satisfaction from workers without a negative impact on task execution time or work quality.

Chatterbox: conversational interfaces for microtask crowdsourcing

This paper proposed a micro-tasking framework that supports skill development of individual workers. Researchers designed the framework based on the review of two micro-tasking projects we have conducted. It consists of four key components, a tutorial generator, a task dispatcher, a feedback visualizer, and a skill assessment engine. The framework will help developers create sustainable micro-tasking systems


Skill Development Framework for Micro-Tasking

Priming has been used in the realms of psychology as a way to alter people's behaviors or performance. It can be overt (supraliminal) or covert (subliminal). A subsequent behavior, which is a result of the priming, occurs typically without conscious awareness. In the field of microtask crowdsourcing, multiple types of primes have been used to increase and alter people's performance.

Priming to Increase Performance in Microtask Crowdsourcing

Algorithmic crowdsourcing concentrates on computational approaches to solving the problems caused by introducing crowdworkers.

Algorithmic Crowdsourcing

Through the use of microtasking, this explores the basics of mircotasking and how under managing microtasking during emergency use can set back projects. 

Microtasking Models and Managerial Challenges 

Having individuals with skill carrying out research and technical tasks for payment. 

Learn Before

Related

Learn After