- Providing workers with a birds-eye view of the overall decomposition is critical.
- Workers may become confused if their role in the larger process is not made explicit, and workers may need to refer to information contained in prior answers to adequately solve their sub-task.
- To communicate the larger goal, and the work completed towards this goal, Turkomatic shows an indented list view of the decomposition generated by other workers, highlighting the current subtask.
- Developers reserve further study of the optimal amount of context information that should be presented to future work.

Crowdsourcing Workflows: Show Context of Tasks in a Workflow

- The HITs for subdivide and merge steps contain complex information and require careful attention of the worker.
- Workers can have difficulty identifying which task in a complex plan they were bring asked to carry out.
- Developers used background colors to emphasize and separate the primary task instructions from explanatory and context information.

Crowdsourcing Workflows: Visually Separate Task Language from Context Language

- Designing general-purpose HITs that fit a wide variety of tasks while still conveying specific requirements of the decomposition and merge phases to a worker is challenging.
- Two design techniques proceed especially useful and necessary: including larger task context in subtasks, and visually separating task language from context language.

Additional Sections:
- Show Context of Tasks in a Workflow
- Visually Separate Task Language from Context Language

Boston College

Turkomatic consists of two major components:
- First, a meta-workflow uses the crowd to assist in the design of the workflow, and to execute it; the price-divide-solve algorithm (PDS) uses a divide-and-conquer strategy to let workers decide how to split up tasks.
- Second, an editable workflow visualization enables requesters to observe and manage work being executed by the crowd.

Additional Sections:
- Algorithmic Model: Price-Divide-Solve, then Merge
- Task Design Strategies
- Requester Interfaces

Crowdsourcing Workflows: The Turkomatic System

Kulkarni, A., Can, M., & Harmann, B. (2012). Collaboratively crowdsourcing workflows with Turkomatic. Association for Computing Machinery, 1003-1012. DOI: https://doi.org/10.1145/2145204.2145354 

Collaboratively crowdsourcing workflows with Turkomatic

- PDS is a general-purpose algorithm for inducing the crowd to generate workflows for tasks specified by the requester.
- PDS guides workers through the process of converting large, complex tasks into microtasks appropriate for crowd markets. 
- Once all the microtasks for a given component of the workflow are solved, a merge stage asks workers to recombine work back into a coherent single solution to the original task.
- Both stages rely on verification functions that determine when work needs to be checked or repeated.
- Turkomatic posts human intelligence tasks (HIT) on Amazon Mechanical Turk, though the algorithm is independent of this particular platform.

Additional Sections:
- Price-Divide-Solve
- Merge Phase
- Verification

Crowdsourcing Workflows: Algorithmic Model: Price-Divide-Solve, then Merge

Crowdsourcing Workflows: Task Design Strategies

This section reviews interaction with Turkomatic from the requester’s perspective.

Additional Sections:
- Requesting Work
- Visualizing Ongoing Work
- Editing Workflows

Learn Before

Related

Learn After