- The use of different custom keyboards have an impact on task execution times, both for single- and multiple-selectin tasks, with statistically significant differences with the text configuration and the code configuration.
- For multiple-selection tasks, the availability of multiple input alternatives yields faster execution times, however no clear total order of performance emerge across the two tasks.
- The removal of button shortcuts has a detrimental effect on workers execution time, while output quality is not affected, which can be attributed to the input validation mechanism implemented in the chatbot that prevents wrong results from being submitted.

Boston College

- The experiments were performed recruiting workers from the F8 microtask crowdsourcing platform.
- Because the goal of the experiment is to understand if text-based conversational agents can enable microtask crowdsourcing, neither quality levels nor qualification were measured.
- 316 distinct workers executed at least one task; 31 workers executed both web and chatbot jobs.

Chatterbox: Evaluation 

Mavridis, P., Huang, O., Qiu, S.. Gadiraju, U., & Bozzon, A. (2019) Chatterbox: conversational interfaces for microtask crowdsourcing. Association for Computing Machinery, 243–251. DOI:https://doi-org.ezproxy.lib.uconn.edu/10.1145/3320435.3320439

Chatterbox: conversational interfaces for microtask crowdsourcing

Sections:
- Execution Time
- Work Quality
- Workers Satisfaction

Chatterbox: RQ1: Standard Web vs Conversational Interfaces

Chatterbox: RQ2: Conversational Interfaces - UI Elements

- Results show that chatbots could be a suitable alternative to Web-based microwork platforms, at least for the considered task types, both in terms of execution time and quality.
- Although a direct comparison is not possible, the results matched the outcome of previous studies with mobile UIs.
- Workers expressed positive opinions about this work interface modality.
- The obtained results are promising, and the researchers’ takeaway from the whole experimental procedure and the results is that the flexibility for the interface to be used, the design of the interface, and the task itself are all important factors to consider when building crowdsourcing tasks for conversational interfaces.
- Researchers argue that the use of conversational interfaces for crowd work can provide a number of potential benefits, such as the further democratization of crowd work, increased workers diversity, and increased workers capacity for low-latency and/or situational microtask crowdsourcing.

Learn Before

Related