- With the exception of the Human OCR task and the Sentiment Analysis task, the execution time distributions for the specific task types have no statistically significant difference.
- Speech Transcription tasks show a slightly longer execution time, which experimenters attribute to the UI design of the Web task.
- The statistically significant difference between the Sentiment Analysis tasks and the Human OCR tasks could be attributed to the presence of long textual instructions at the beginning of the conversational interface.
- On 84 occasions, workers used the task reviewing and editing functionality to correct their answers before submitting the results.

Chatterbox: Execution Time

- Researchers observe comparable performance across tasks with precision that is slightly lower with Chatbot tasks.
- A manual analysis of the results highlights an interesting difference with the Human OCR tasks, where errors were mostly due to ambiguous characters in the CAPTCHA, but less present with chatbot workers.

Chatterbox: Work Quality

- Workers participating in Chatbot tasks were also asked to provide feedback on their experience with microwork executed through conversational interfaces.
- 349 of 600 task executions received comment; 81.9% of comments were positive opinions, 44 workers gave a neutral comment, and 19 workers gave a negative opinion about the slow response of the chatbot.
- The majority of comments highlighted the intuitive user experience, and others remarked that it was an enjoyable work experience.

Chatterbox: Workers Satisfaction

Sections:
- Execution Time
- Work Quality
- Workers Satisfaction

Boston College

- The experiments were performed recruiting workers from the F8 microtask crowdsourcing platform.
- Because the goal of the experiment is to understand if text-based conversational agents can enable microtask crowdsourcing, neither quality levels nor qualification were measured.
- 316 distinct workers executed at least one task; 31 workers executed both web and chatbot jobs.

Chatterbox: Evaluation 

Mavridis, P., Huang, O., Qiu, S.. Gadiraju, U., & Bozzon, A. (2019) Chatterbox: conversational interfaces for microtask crowdsourcing. Association for Computing Machinery, 243–251. DOI:https://doi-org.ezproxy.lib.uconn.edu/10.1145/3320435.3320439

Chatterbox: conversational interfaces for microtask crowdsourcing

Chatterbox: RQ1: Standard Web vs Conversational Interfaces

- The use of different custom keyboards have an impact on task execution times, both for single- and multiple-selectin tasks, with statistically significant differences with the text configuration and the code configuration.
- For multiple-selection tasks, the availability of multiple input alternatives yields faster execution times, however no clear total order of performance emerge across the two tasks.
- The removal of button shortcuts has a detrimental effect on workers execution time, while output quality is not affected, which can be attributed to the input validation mechanism implemented in the chatbot that prevents wrong results from being submitted.

Chatterbox: RQ2: Conversational Interfaces - UI Elements

- Results show that chatbots could be a suitable alternative to Web-based microwork platforms, at least for the considered task types, both in terms of execution time and quality.
- Although a direct comparison is not possible, the results matched the outcome of previous studies with mobile UIs.
- Workers expressed positive opinions about this work interface modality.
- The obtained results are promising, and the researchers’ takeaway from the whole experimental procedure and the results is that the flexibility for the interface to be used, the design of the interface, and the task itself are all important factors to consider when building crowdsourcing tasks for conversational interfaces.
- Researchers argue that the use of conversational interfaces for crowd work can provide a number of potential benefits, such as the further democratization of crowd work, increased workers diversity, and increased workers capacity for low-latency and/or situational microtask crowdsourcing.

Learn Before

Related

Learn After