- Researchers observe comparable performance across tasks with precision that is slightly lower with Chatbot tasks.
- A manual analysis of the results highlights an interesting difference with the Human OCR tasks, where errors were mostly due to ambiguous characters in the CAPTCHA, but less present with chatbot workers.

Boston College

Sections:
- Execution Time
- Work Quality
- Workers Satisfaction

Chatterbox: RQ1: Standard Web vs Conversational Interfaces

Mavridis, P., Huang, O., Qiu, S.. Gadiraju, U., & Bozzon, A. (2019) Chatterbox: conversational interfaces for microtask crowdsourcing. Association for Computing Machinery, 243–251. DOI:https://doi-org.ezproxy.lib.uconn.edu/10.1145/3320435.3320439

Chatterbox: conversational interfaces for microtask crowdsourcing

- With the exception of the Human OCR task and the Sentiment Analysis task, the execution time distributions for the specific task types have no statistically significant difference.
- Speech Transcription tasks show a slightly longer execution time, which experimenters attribute to the UI design of the Web task.
- The statistically significant difference between the Sentiment Analysis tasks and the Human OCR tasks could be attributed to the presence of long textual instructions at the beginning of the conversational interface.
- On 84 occasions, workers used the task reviewing and editing functionality to correct their answers before submitting the results.

Chatterbox: Execution Time

Chatterbox: Work Quality

- Workers participating in Chatbot tasks were also asked to provide feedback on their experience with microwork executed through conversational interfaces.
- 349 of 600 task executions received comment; 81.9% of comments were positive opinions, 44 workers gave a neutral comment, and 19 workers gave a negative opinion about the slow response of the chatbot.
- The majority of comments highlighted the intuitive user experience, and others remarked that it was an enjoyable work experience.

Learn Before

Related