Crowdsourcing Workflows: Table 1: Experimental Results

- One class of successes were snap judgements: cases where there was no decomposition of the input task, but where a worker marked the first task as solvable, and a subsequent worker provided a correct, coherent, and complete answer.
- In these outcomes, a correct result is produced without additional decomposition or planning.
- This result is unsurprising, as no collaboration is involved when no workflow is required: the same worker both determines how to solve a task and solves it.

Crowdsourcing Workflows: Snap Judgements

- In some cases, groups of workers broke down tasks in line with requester intentions, and this was sustained over multiple iterations.
- This outcome occurred infrequently, which is not surprising to researchers; it required a shared cognitive model of how work should be decomposed to be maintained by multiple workers in the execution process.
- Researchers found that active participation of one or more highly eager workers early in the princess made a substantial difference in downstream quality.

Crowdsourcing Workflows: Successful Planning

- In the majority of cases, the unsupervised crowd produced unsuitable workflows, or unsuccessful results.
- The most frequent type of failure was derailment, a phenomenon that occurred when the PDS algorithm failed to terminate and continued to produce steps indefinitely.

Derailment can happen for multiple reasons: 
- First, workers were confused about appropriate task granularity.
- Second, workers authored subdivisions that could be executed by a single worker, but not split across separate workers.
- Finally, workers who had lost the context of the overall workflow generated decompositions that restated previous tasks as subtasks, leading to cyclic behavior.

Crowdsourcing Workflows: Challenges: Task Derailment, Emergent Complexity, and Cycling

- Some workflows failed to complete due to starvation; after a while, no new workers attempted the available tasks, and the time limit for the experiment expired without the execution of work continuing further.
- Task starvation has been observed in other projects, and has been counteracted thought listing optimization or chaining sets of tasks to retain workers.
- In the context of the PDS algorithm, starvation occurred most often when a worker marked a task as solvable when, judging by its complexity, it should have been subdivided.

Crowdsourcing Workflows: Task Starvation

- Adding redundancy by asking more workers to contribute and vote on subdivisions at each step failed to yield more successful decomposition.
- One explanation for this result is that Turkomatic’s model for redundancy does not allow workers to iterate towards correct answers, but simple increases the parallelism in each step.
- Increasingly, eager workers sometimes arrived in already derailed workflows and made efforts to correct the state of the system.
- In several cases, workers accepted Turkomatic HITs, but were not satisfied with the workflow provided by previous members of the crowd, and these workers chose to email the authors directly to suggest improvements to the task.
- This result suggests that making Turkomatic requester interface available to motivated workers to allow the crowd to self-police and improve its own work would be more effective.

Crowdsourcing Workflows: Standard Quality Assurance Techniques are Insufficient

Researchers observed both successes and systematic failures.

Additional Sections:
- Table 1: Experimental Results
- Snap Judgements
- Successful Planning
- Challenges: Task Derailment, Emergent Complexity, and Cycling
- Task Starvation
- Standard Quality Techniques are Insufficient

Boston College

To explore how effectively crowds can be used to support the execution of complex work, researchers performed two evaluations:
- First, researchers examined how crowds performed in producing and solving workflows without expert intervention.
- Second, researchers looked at how expert intervention can improve the crowd’s performance.

Additional Sections:
- Unsupervised Crowd Planning
- Procedure
- Results and Observations

Crowdsourcing Workflows: How Well Can Crowds Plan and Execute?

Kulkarni, A., Can, M., & Harmann, B. (2012). Collaboratively crowdsourcing workflows with Turkomatic. Association for Computing Machinery, 1003-1012. DOI: https://doi.org/10.1145/2145204.2145354 

Collaboratively crowdsourcing workflows with Turkomatic

- Researchers ask the following question: can the crowd be guided algorithmically to plan and solve problems without any input from requesters?
- Undirected crowd planning seems unlikely to produce usable workflows; design problems often require extensive communication and coordination between the requester and worker, and correct interpretation of a prompt may rely on domain specific knowledge.
- To provide a baseline of success and failure patterns, researchers ran the PDS algorithm with no monitoring by the requester.

Crowdsourcing Workflows: Unsupervised Crowd Planning

- Via Turkomatic, researchers requester that the crowd plan workflows for and compute answers to several types of objectives, including essay writing, natural language queries, itinerary planning, Java programming, and multimedia content generation.
- Researchers ran more than 20 distinct kinds of queries in total, where each HIT earned the worker a different price.
- Workers were given 20 minutes to complete each HIT.

Requests given to the crowd were phrased as follows:
- Essay writing: “write a 3-paragraph essay about crowdsourcing.”
- Natural language query: “create a list of the names of Department Chairs of the top 20 computer science college programs in the US.”
- Itinerary planning: “plan and complete a road trip from San Francisco, California, to New York City. Completely include the locations of all necessary hotels, restaurants, and sights along the way.”
- Java programming: “Please write a short piece of Java code to reverse a string. The algorithm should take as input a string and output its reverse. Make sure it compiles.”

Learn Before

Related

Learn After