- Requesters post new Turkomatic jobs through a natural-language web interface; it offers a single text box where requesters specify what they want to accomplish.
- Expert requesters can also author decompositions that serve as a starting point for the PDS algorithm, which may be preferred if the requester believes that crowds will not be able to properly decompose a given task.
- Once a new task prompt or task tree is loaded, the PDS solve the leaf tasks, and eventually merge solutions to accomplish the root tasks.

Crowdsourcing Workflows: Requesting Work

- To enable requesters to gain insight into partially completed work, Turkomatic provides a workflow visualization that shows the current state of an ongoing job as either an indented list or a decomposition graph.
- This list view allocates the most screen space to task content; the graph also shows hierarchy, as well as parallel and serial decompositions.
- Both visualizations inform requesters how much of the work has been accomplished, what strategies have been taken, and whether subtask solutions or decompositions are of sufficient quality.

Crowdsourcing Workflows: Visualizing Ongoing Work

- Ongoing work can be unsatisfactory for multiple reasons: a crowd-authored decomposition may be flawed, or a solution to a subtask may be of low quality.
- Requesters can edit existing workflows in real-time to address such challenges.
- Requesters edit the task description, decomposition, or solution for any one node; once an edit is completed, Turkomatic computes which tasks will have to be performed again by additional workers.
- When a task description is changed, any subtasks created for this task by the crowd may also no longer be valid, so Turkomatic then invalidates the entire subtree below the edited task and reissues the tasks.
- When a requester edits a task solution or decomposition directly, the entire subtree of that task is discarded.

Crowdsourcing Workflows: Editing Workflows

This section reviews interaction with Turkomatic from the requester’s perspective.

Additional Sections:
- Requesting Work
- Visualizing Ongoing Work
- Editing Workflows

Boston College

Turkomatic consists of two major components:
- First, a meta-workflow uses the crowd to assist in the design of the workflow, and to execute it; the price-divide-solve algorithm (PDS) uses a divide-and-conquer strategy to let workers decide how to split up tasks.
- Second, an editable workflow visualization enables requesters to observe and manage work being executed by the crowd.

Additional Sections:
- Algorithmic Model: Price-Divide-Solve, then Merge
- Task Design Strategies
- Requester Interfaces

Crowdsourcing Workflows: The Turkomatic System

Kulkarni, A., Can, M., & Harmann, B. (2012). Collaboratively crowdsourcing workflows with Turkomatic. Association for Computing Machinery, 1003-1012. DOI: https://doi.org/10.1145/2145204.2145354 

Collaboratively crowdsourcing workflows with Turkomatic

- PDS is a general-purpose algorithm for inducing the crowd to generate workflows for tasks specified by the requester.
- PDS guides workers through the process of converting large, complex tasks into microtasks appropriate for crowd markets. 
- Once all the microtasks for a given component of the workflow are solved, a merge stage asks workers to recombine work back into a coherent single solution to the original task.
- Both stages rely on verification functions that determine when work needs to be checked or repeated.
- Turkomatic posts human intelligence tasks (HIT) on Amazon Mechanical Turk, though the algorithm is independent of this particular platform.

Additional Sections:
- Price-Divide-Solve
- Merge Phase
- Verification

Crowdsourcing Workflows: Algorithmic Model: Price-Divide-Solve, then Merge

- Designing general-purpose HITs that fit a wide variety of tasks while still conveying specific requirements of the decomposition and merge phases to a worker is challenging.
- Two design techniques proceed especially useful and necessary: including larger task context in subtasks, and visually separating task language from context language.

Additional Sections:
- Show Context of Tasks in a Workflow
- Visually Separate Task Language from Context Language

Learn Before

Related

Learn After