Each participant’s submission was classified into “good,” “fair,” and “poor” categories based on the three images (same as phase 1) produced from the simulation. The crowd workers performed similarly to the graduate students, and even produced more “good” simulations than one group of graduate students. The percentages of submissions in each category were as follows:

- Crowd workers: 57% good, 21% fair, 21% poor
- University One Students: 47% good, 24% fair, 29% poor
- University Two students: 62% good, 38% fair, 0% poor

University of Texas at Austin

The simulations were assessed based on the accuracy and validity of the simulation results. The participants’ results were analyzed based on how many simulation trials they ran and the amount of time they spent logged into the simulation. 

Phase 2 Results & Analysis (Crowdsourcing Complex Engineering Tasks)

Staffelbach M, Sempolinski P, Kijewski-Correa T, Thain D, Wei D, Kareem A, et al. (2015). Lessons Learned from Crowdsourcing Complex Engineering Tasks. PLoS ONE 10(9): e0134978. https://doi.org/10.1371/journal.pone.0134978


Lessons Learned from Crowdsourcing Complex Engineering Tasks

The effort made in this phase varied between groups. The median number of simulations ran before completion overall was 8 trials across all groups, with a few outliers that skewed the averages of each group. The average number of simulations ran before completion in each group was as follows:

- Crowd workers: 28.9
- University One students: 12.2
- University Two students: 18.2

Number of Simulation Trials (Crowdsourcing Complex Engineering Tasks)

The time spent logged into the simulation was also an indicator of effort made. The median amount of time logged in overall was 5 hours across all groups, with a few outliers skewing the overall average to 9.4 hours.

Learn Before

Related