- Researchers ask the following question: can the crowd be guided algorithmically to plan and solve problems without any input from requesters?
- Undirected crowd planning seems unlikely to produce usable workflows; design problems often require extensive communication and coordination between the requester and worker, and correct interpretation of a prompt may rely on domain specific knowledge.
- To provide a baseline of success and failure patterns, researchers ran the PDS algorithm with no monitoring by the requester.

Boston College

To explore how effectively crowds can be used to support the execution of complex work, researchers performed two evaluations:
- First, researchers examined how crowds performed in producing and solving workflows without expert intervention.
- Second, researchers looked at how expert intervention can improve the crowd’s performance.

Additional Sections:
- Unsupervised Crowd Planning
- Procedure
- Results and Observations

Crowdsourcing Workflows: How Well Can Crowds Plan and Execute?

Kulkarni, A., Can, M., & Harmann, B. (2012). Collaboratively crowdsourcing workflows with Turkomatic. Association for Computing Machinery, 1003-1012. DOI: https://doi.org/10.1145/2145204.2145354 

Collaboratively crowdsourcing workflows with Turkomatic

Crowdsourcing Workflows: Unsupervised Crowd Planning

- Via Turkomatic, researchers requester that the crowd plan workflows for and compute answers to several types of objectives, including essay writing, natural language queries, itinerary planning, Java programming, and multimedia content generation.
- Researchers ran more than 20 distinct kinds of queries in total, where each HIT earned the worker a different price.
- Workers were given 20 minutes to complete each HIT.

Requests given to the crowd were phrased as follows:
- Essay writing: “write a 3-paragraph essay about crowdsourcing.”
- Natural language query: “create a list of the names of Department Chairs of the top 20 computer science college programs in the US.”
- Itinerary planning: “plan and complete a road trip from San Francisco, California, to New York City. Completely include the locations of all necessary hotels, restaurants, and sights along the way.”
- Java programming: “Please write a short piece of Java code to reverse a string. The algorithm should take as input a string and output its reverse. Make sure it compiles.”

Crowdsourcing Workflows: Procedure

Researchers observed both successes and systematic failures.

Additional Sections:
- Table 1: Experimental Results
- Snap Judgements
- Successful Planning
- Challenges: Task Derailment, Emergent Complexity, and Cycling
- Task Starvation
- Standard Quality Techniques are Insufficient

Learn Before

Related