Sections:
- Execution Time
- Work Quality
- Workers Satisfaction

Chatterbox: RQ1: Standard Web vs Conversational Interfaces

- The use of different custom keyboards have an impact on task execution times, both for single- and multiple-selectin tasks, with statistically significant differences with the text configuration and the code configuration.
- For multiple-selection tasks, the availability of multiple input alternatives yields faster execution times, however no clear total order of performance emerge across the two tasks.
- The removal of button shortcuts has a detrimental effect on workers execution time, while output quality is not affected, which can be attributed to the input validation mechanism implemented in the chatbot that prevents wrong results from being submitted.

Chatterbox: RQ2: Conversational Interfaces - UI Elements

- Results show that chatbots could be a suitable alternative to Web-based microwork platforms, at least for the considered task types, both in terms of execution time and quality.
- Although a direct comparison is not possible, the results matched the outcome of previous studies with mobile UIs.
- Workers expressed positive opinions about this work interface modality.
- The obtained results are promising, and the researchers’ takeaway from the whole experimental procedure and the results is that the flexibility for the interface to be used, the design of the interface, and the task itself are all important factors to consider when building crowdsourcing tasks for conversational interfaces.
- Researchers argue that the use of conversational interfaces for crowd work can provide a number of potential benefits, such as the further democratization of crowd work, increased workers diversity, and increased workers capacity for low-latency and/or situational microtask crowdsourcing.

Chatterbox: Discussion and Implications

- The experiments were performed recruiting workers from the F8 microtask crowdsourcing platform.
- Because the goal of the experiment is to understand if text-based conversational agents can enable microtask crowdsourcing, neither quality levels nor qualification were measured.
- 316 distinct workers executed at least one task; 31 workers executed both web and chatbot jobs.

Boston College

Conversational interfaces can facilitate human-computer interactions, but researchers still speculate as to if conversational interfaces can improve worker experience and work quality in crowdsourcing marketplaces. Researchers explore this concept by designing Chatterbox, aimed at gauging the interest and acceptance by crowdworkers for this type of interface. Results suggest that conversational interfaces can be used effectively for crowdsourcing microtasks, resulting in high satisfaction from workers without a negative impact on task execution time or work quality.

Chatterbox: conversational interfaces for microtask crowdsourcing

Mavridis, P., Huang, O., Qiu, S.. Gadiraju, U., & Bozzon, A. (2019) Chatterbox: conversational interfaces for microtask crowdsourcing. Association for Computing Machinery, 243–251. DOI:https://doi-org.ezproxy.lib.uconn.edu/10.1145/3320435.3320439

- The availability and usage of messaging applications (such as Telegram, Facebook Messenger, etc.), as well as their popularity and recent advances in machine learning capabilities, has sparked new interest in conversational interfaces and chatbots.
- The crowing popularity of conversational interfaces has coincided with flourishing crowdsourcing marketplaces.
- Microtask crowdsourcing allows the interaction with a large crowd of diverse people for data processing or analysis purposes.
- Microtask crowdsourcing is commonly executed by means of dedicated Web platforms, where all the published microtasks are publicly presented to workers, and upon selection of the task, users are directed to a different webpage that the task requires.
- Recent work has shed light on the importance of task design choices made with respect to user interface elements; and on how such choices can influence the quality of work produced and satisfaction among workers.
- Although conversational interfaces have been effectively used in numerous applications, the impact of conversational interfaces in microtask crowdsourcing marketplaces has remained unexplored.
- Lowering the entry barrier for workers to participate effectively in crowdsourcing tasks is an important step towards securing the future of crowd work.
- The availability of effective automated text-based conversational interfaces as an alternative to the traditional Web UI, could broaden the pool of available crowd workers by easing their unfamiliarity with the interface elements.

Chatterbox: Introduction

- Conversational agent definition: a software programmed to automatically interpret and respond to requests expressed in natural language, so to mimic the behavior of a human interlocutor.
- Chatbots definition: a class of conversational agents that prevalently use text as an interaction medium.
- While research on chatbot systems dates back to the 1960s, the growing popularity of messaging platforms has sparked new interest both in industry and academia.

Chatterbox: Background and Related Work

Researchers put together a list of 5 types of microtasks, typically completed by crowd workers in microwork crowdsourcing marketplaces, selected to stress the diversity of evaluated content types and the diversity of UI elements used to perform the tasks.

Chatterbox: Experimental Design

Chatterbox: Evaluation 

- Test-based conversational agents are witnessing widespread adoption as effective tools to automate repetitive tasks, or as an alternative to traditional information seeking interfaces.
- In this paper, researchers provide evidences of their suitability as microtask crowdsourcing platform.
- Researchers highlighted the importance of task-specific interaction design, but also the convenience of advanced text input interfaces currently available in messaging platforms.
- This work provides plenty of inspirations for future research directs, but more research is needed to better understand the peak performance achievable with different task and content types.
- Researchers are also interested in investigating the utility and performance conversational interfaces addressed to requester, both for task creation and monitoring.

Learn Before

Related

Learn After