The effort made in this phase varied between groups. The median number of simulations ran before completion overall was 8 trials across all groups, with a few outliers that skewed the averages of each group. The average number of simulations ran before completion in each group was as follows:

- Crowd workers: 28.9
- University One students: 12.2
- University Two students: 18.2

Number of Simulation Trials (Crowdsourcing Complex Engineering Tasks)

The time spent logged into the simulation was also an indicator of effort made. The median amount of time logged in overall was 5 hours across all groups, with a few outliers skewing the overall average to 9.4 hours.

Time Spent Logged In (Crowdsourcing Complex Engineering Tasks)

Each participant’s submission was classified into “good,” “fair,” and “poor” categories based on the three images (same as phase 1) produced from the simulation. The crowd workers performed similarly to the graduate students, and even produced more “good” simulations than one group of graduate students. The percentages of submissions in each category were as follows:

- Crowd workers: 57% good, 21% fair, 21% poor
- University One Students: 47% good, 24% fair, 29% poor
- University Two students: 62% good, 38% fair, 0% poor

Participant Scores (Crowdsourcing Complex Engineering Tasks)

The simulations were assessed based on the accuracy and validity of the simulation results. The participants’ results were analyzed based on how many simulation trials they ran and the amount of time they spent logged into the simulation. 

University of Texas at Austin

A follow-up experiment was conducted to assess the ability of crowd workers to create simulations. In this phase, individuals in both the crowd worker group and the graduate student group were given the same complex task of creating a simulation on the Virtual Wind Tunnel website. The participants were given a building to analyze and were instructed to perform a simulation on it to the best of their ability. They were allowed to change some of the simulation parameters and expected to generate valid, high-quality results using the knowledge and skills they acquired in Phase 1. The participants were compensated for their work, and were also motivated by a cash prize for the best simulation within each experimental group.

Phase 2 (Crowdsourcing Complex Engineering Tasks)

Staffelbach M, Sempolinski P, Kijewski-Correa T, Thain D, Wei D, Kareem A, et al. (2015). Lessons Learned from Crowdsourcing Complex Engineering Tasks. PLoS ONE 10(9): e0134978. https://doi.org/10.1371/journal.pone.0134978


Lessons Learned from Crowdsourcing Complex Engineering Tasks

Phase 2 Results & Analysis (Crowdsourcing Complex Engineering Tasks)

This follow-up experiment went beyond the typical limits of what crowdsourcing tasks expect workers to do, as most crowdsourcing involves simple tasks that can be completed in short amounts of time. However, this phase involved repeated experimentation and analysis from the participants, which more closely fits the description of outsourcing (seeking work from an external source instead of an internal source) rather than crowdsourcing. This practice of outsourcing had two consequences: there was a much lower participation rate than phase 1, and the workers varied widely on the amount of time they were willing to spend with the simulation. 

Learn Before

Related

Learn After