Between the two groups, there was a significant difference between the scores of the crowd workers and graduate students. The crowd workers produced more accurate answers to the questions in the tasks. A two-sample t-test showed a statistical difference with a p-value of 2.339x10^-5.

University of Texas at Austin

The two experimental groups were further divided into sub-groups for analysis. The crowd workers were grouped into workers with a masters qualification from Amazon Mechanical Turk (Masters), and workers without the masters qualification (non-Masters). The graduate students were grouped based on the university they went to, as the researchers recruited them from two universities: one in the US and one in China (universities One and Two). However, the main focus of the study was to compare the crowd workers to the graduate students, so these sub-groups were distinguished for analysis purposes only.

Phase 1 Results & Analysis (Crowdsourcing Complex Engineering Tasks)

Staffelbach M, Sempolinski P, Kijewski-Correa T, Thain D, Wei D, Kareem A, et al. (2015). Lessons Learned from Crowdsourcing Complex Engineering Tasks. PLoS ONE 10(9): e0134978. https://doi.org/10.1371/journal.pone.0134978


Lessons Learned from Crowdsourcing Complex Engineering Tasks

In order to determine ground truth, the researchers recruited an expert in Computational Fluid Dynamics (CFD), the branch of mechanics that the Virtual Wind Tunnel simulation was based on. This expert answered the same questions given to the participants, and the expert’s responses were used as a basis for right answers (ground truth). The participants’ responses were compared to the responses of the expert, and each participant’s score was determined based on the percentage agreement with the ground truth. The average score breakdown of the groups was as follows:

- Master crowd workers: 79.79%
- Non-Master crowd workers: 78.25%
- University One students: 67.91%
- University Two students: 69.04%

Average Participant Scores (Crowdsourcing Complex Engineering Tasks)

Each sub-group had a majority consensus calculated within the group. The average percent agreement with the majority consensus in each group was as follows:

- Master crowd workers: 85.46%
- Non-Master crowd workers: 82.97%
- University One students: 82.06%
- University Two students: 78.56%

Learn Before

Related