- When requesters used workflow editing tools to guide the crowd’s efforts, tasks were completed successfully; intervention enabled requesters to provide feedback and to iterate on unsuccessful tasks.
- If a workflow is executed without input from the requester, workers must design plans or create content with only a limited understanding of the requester’s intent or preferences, which is in striking contrast to the traditional model of design where the designer and client converge on a solution together.
- For complex work, it is important for the requester to provide feedback, and manual intervention and editing of a crowd-generated workflow is an effective way to do so.

Boston College

- The results of the previous section suggest that the crowd can sometimes produce acceptable results in planning and executing solutions to general requests, but more frequently face challenges.
- This section elaborates more on informal experiments for both approaches, first with an expert crowd of workers recruited at UC Berkeley, and the second with a Mechanical Turk crowd.
- Researchers re-ran the essay-writing, blog creation, and natural-language query tasks discussed earlier.

At least two different strategies to address these challenges are plausible:
- The first strategy is to recruit more expert workers.
- The second strategy is to allow requesters to monitor and selectively intervene in the process.

Additional Sections:
- Outcome
- Discussion: Why Does Collaborative Workflow Creation Succeed?

Crowdsourcing Workflows: Collaborative Planning and Execution

Kulkarni, A., Can, M., & Harmann, B. (2012). Collaboratively crowdsourcing workflows with Turkomatic. Association for Computing Machinery, 1003-1012. DOI: https://doi.org/10.1145/2145204.2145354 

Collaboratively crowdsourcing workflows with Turkomatic

- Both conditions avoided starvation and derailment, and resulted in usable solutions. 
- When expert workers carried out decomposition and solution of the subtasks, solutions were reached rapidly.
- When requesters intervened using the workflow editor, the resulting workflows were executed correctly in all cases, meaning they reached a correct solution without derailment or starvation.

Additional Sections:
- Table 2: Experimental Results

Learn Before

Related