This experiment assessed the limitations of crowdsourcing as a method of completing difficult, complex tasks. The researchers aimed to find out how large groups of untrained crowd workers could be used to work on civil engineering tasks in order to expand the available workforce beyond just specialized engineers. This would increase overall productivity in the formal workplace by allowing certain tasks to be designated to crowd workers. As a result, crowd work could be utilized in specific situations in which there must be a large amount of time-sensitive work done, such as rebuilding infrastructure after a natural disaster.

Introduction (Crowdsourcing Complex Engineering Tasks)

Both the crowd workers and the civil engineering graduate students completed tasks that contained survey questions about Virtual Wind Tunnel data. The tasks instructed the participants to analyze and evaluate graphs containing the data. Since the graduate students had adequate expertise and sufficient civil engineering knowledge, they served as a control group for comparison with the crowd workers. The students were not familiar with the particular tasks, but rather were able to use their civil engineering knowledge to aid their completion of the tasks. Meanwhile, the crowd workers completed these tasks without the expertise of a civil engineer, which allowed the researchers to gain insight into the workers. The crowd workers had different levels of qualification regarding their contributions within Amazon Mechanical Turk (i.e. approval ratings, number of tasks previously completed with Amazon Mechanical Turk).

Phase 1 (Crowdsourcing Complex Engineering Tasks)

A follow-up experiment was conducted to assess the ability of crowd workers to create simulations. In this phase, individuals in both the crowd worker group and the graduate student group were given the same complex task of creating a simulation on the Virtual Wind Tunnel website. The participants were given a building to analyze and were instructed to perform a simulation on it to the best of their ability. They were allowed to change some of the simulation parameters and expected to generate valid, high-quality results using the knowledge and skills they acquired in Phase 1. The participants were compensated for their work, and were also motivated by a cash prize for the best simulation within each experimental group.

Phase 2 (Crowdsourcing Complex Engineering Tasks)

Crowd workers are able to complete complex tasks to a similar ability as trained experts, as long as they are provided with descriptive tutorials and adequate compensation. The more complex the tasks are, the more the crowd workers will communicate with the researchers and experts-- which is more similar to outsourcing rather than crowdsourcing. This experiment encourages the use of crowd workers to be trainable assistants to check over the work of experts. If complex tasks (such as data analysis or surveys) have adequate tutorials to train crowd workers with little expertise, then these tasks can be completed by those crowd workers efficiently and accurately in order to increase workplace efficiency. However, if crowd workers are asked to use unfamiliar or complicated methods or tools to complete the tasks, this would require increased communication between the crowd workers and experts which would decrease efficiency in the workplace. Therefore, if complex tasks are assigned to crowd workers, workplace efficiency can be increased as long as there are adequate tutorials and compensation in place.

Conclusions (Crowdsourcing Complex Engineering Tasks)

This study was conducted to explore the limits of crowdsourcing using complex technical engineering tasks. Two populations were analyzed in this experiment: anonymous crowd workers recruited from Amazon Mechanical Turk (a marketplace for crowdsourcing microtasks) and civil engineering graduate students. The completion of the tasks was compared between the two groups to assess the effectiveness of using untrained crowd workers for complex tasks that required expertise.

University of Texas at Austin

Citizen science is research contributed by nonprofessional participants via crowdsourcing. In citizen science projects, participants either collect data with mobile technologies or analyze existing data through internet-based Virtual Citizen Science (VCS) platforms, while in the accompaniment of a professional scientist or researcher.
This research improves the scientific community's capacity and increases the public's understanding of science.

Citizen science

Staffelbach M, Sempolinski P, Kijewski-Correa T, Thain D, Wei D, Kareem A, et al. (2015). Lessons Learned from Crowdsourcing Complex Engineering Tasks. PLoS ONE 10(9): e0134978. https://doi.org/10.1371/journal.pone.0134978


Lessons Learned from Crowdsourcing Complex Engineering Tasks

Bonney, R., Shirk, J. L., Phillips, T. B., Wiggins, A., Ballard, H. L., Miller-Rushing, A. J., & Parrish, J. K. (2014). Next steps for citizen science. Science, 343(6178), 1436–1437. https://doi.org/10.1126/science.1251554

Next Steps for Citizen Science

A new dawn for citizen science

Terminology usage and understanding impacts how knowledge is developed. Defining terms for understanding and applying citizen science can help with broad application and general understanding of the science.

The Importance of Terminology in Citizen Science

Direct communication with researchers and participants is important, as well as review one's understanding of the terminology being used in Citizen Science studies. There needs to be a recognition that one term is usually not applicable to all contexts. Further studies on the use of terminology or the review of it in citizen Science is recommended.

Lessons Learned from the Importance of Terminology in Citizen Science

Mueller, M., Tippins, D., Bryan, L., The Future of Citizen Science. Democracy & Education, vol 20, No 1. https://democracyeducationjournal.org/cgi/viewcontent.cgi?article=1026&context=home

The Future of Citizen Science

- community science
- crowd science
- crowd-sourced science
- civic science
- volunteer monitoring
Public Participation in Scientific Research (PPSR)

Alternative names for Citizen Science

This article focuses on the potential contributions of volunteers to scientific research on wildlife in their local communities, such as research on the mammals living along the Appalachian Trail. It also notes the increasing use of citizen scientists across studies and their critical impact on the advancement of ecological research and findings.

Citizen Science: Can Volunteers Do Real Research?

There are research gaps on citizen science in education. In addition, there is a lack of citizen social science projects. Citizen science is also mainly done by adults. Lastly, there is a lack of records of youth participation in citizen science in a school context. 


Gaps in Research on Youth Participation in Citizen Science 

Studies have shown that there have been positive educational outcomes because of youth participation in citizen science. As a result, educators want to promote citizen science projects in the educational system. This study took place in an Austrian school where students participated in projects that dissected what kinds of political topics young people encounter in their everyday lives. 

Citizen Science in Schools

Sexual and reproductive health research currently lacks:
- Collaboration that targets the gaps in evidence
- High quality tools for patients and the public to participate
- Patients and the public as stakeholders, not just as research subject, to create incentive and recognize their contributions

The Need for Citizen Science in Sexual and Reproductive Health Research 

The article examines how would citizen science develop when coupled with emerging technologies, in terms of its research processes, program and participant cultures, and scientific communities. 

The future of citizen science: emerging technologies and shifting paradigms

This Dutch study explores the question of whether citizen science can be effectively used in decision making regarding public health policies. To this end, challenges that come with incorporating citizen science into public health are discussed, as well as ways these issues can be remedied. Finally, a framework for using citizen science in public health is suggested.

Citizen Science for public health

Citizen science projects that utilize technology and are predominantly worked online are considered to be citizen cyberscience. Some citizen science projects strictly collect data in a physical manner, restricting them from falling into the subset of cyberscience. 

Learn Before

Related

Learn After