- Information finding: workers are tasked to find specific relevant information from a given data source, comprised of 50 tasks from 17 business records.
- Human OCR (CAPTCHA): workers are tasked with transcribing the text contained in a CAPTCHA image, comprised of 50 distinct CAPTCHAS of 4 characters.
- Speech transcription: workers are tasked with transcribing recordings of English speech retrieved from Tatoeba, comprised of 50 distinct recordings with length ranging from 2 to 8 seconds.
- Sentiment analysis: workers are tasked with assessing the sentiment of user reviews, comprised of 50 reviews with the overall sentiments reading as positive, negative, or neutral.
- Image annotation: workers are tasked with determining the categories of food items contained in an image, comprised of 50 distinct images, with categories such as eggs, fish, meat, vegetables, fruits, cheese, mushroom, grain, sweets, and non-food.

Chatterbox: Task Types

- Researchers focused on three types of UI elements that are required to perform the task types investigated in the experiment, including “free text,” “single selection from list,” and “multiple selection from list.”
- Table 1 provides a summary of considered UI elements and their implementation in web and conversational interfaces.

Chatterbox: Work Interfaces

- To answer RQ1, researchers designed 12 experimental conditions, with working interface type (Web, conversational) and task type as independent variables.
- Instructions at the beginning of the conversational tasks are relatively long, thus possibly affecting task execution time, and to account for this, researchers include 6 additional experimental conditions where the conversational interface has task instructions partially hidden, and workers could instruct the chatbot through specific commands to display more detailed instructions.
- With RQ2, researchers tested 3 custom keyboard configurations with the sentiment analysis and image annotation tasks, thus adding 6 additional experimental conditions.

Chatterbox: Experimental Conditions

On the FigureEight (F8) platform, researchers set up two types of tasks: 
- Web jobs
- Conversational jobs.

Chatterbox: Task Assignment and Execution

- The dependent variables in this experiment include execution time, answer quality, and workers satisfaction.
- Execution time is measured as the time (in seconds) between the start and submission of the task.
- Answer quality is measured by comparing the worker answers with ground truth sentiment analysis and image annotation; information finding and speech transcription tasks were inspected manually; human OCR task answers were compared to the label of the CAPTCHA.
- Worker satisfaction of both web and chatbot tasks is measured by default task ratings on F8 after workers finish the task.

Chatterbox: Evaluation Metrics

Researchers put together a list of 5 types of microtasks, typically completed by crowd workers in microwork crowdsourcing marketplaces, selected to stress the diversity of evaluated content types and the diversity of UI elements used to perform the tasks.

Boston College

Conversational interfaces can facilitate human-computer interactions, but researchers still speculate as to if conversational interfaces can improve worker experience and work quality in crowdsourcing marketplaces. Researchers explore this concept by designing Chatterbox, aimed at gauging the interest and acceptance by crowdworkers for this type of interface. Results suggest that conversational interfaces can be used effectively for crowdsourcing microtasks, resulting in high satisfaction from workers without a negative impact on task execution time or work quality.

Chatterbox: conversational interfaces for microtask crowdsourcing

Mavridis, P., Huang, O., Qiu, S.. Gadiraju, U., & Bozzon, A. (2019) Chatterbox: conversational interfaces for microtask crowdsourcing. Association for Computing Machinery, 243–251. DOI:https://doi-org.ezproxy.lib.uconn.edu/10.1145/3320435.3320439

- The availability and usage of messaging applications (such as Telegram, Facebook Messenger, etc.), as well as their popularity and recent advances in machine learning capabilities, has sparked new interest in conversational interfaces and chatbots.
- The crowing popularity of conversational interfaces has coincided with flourishing crowdsourcing marketplaces.
- Microtask crowdsourcing allows the interaction with a large crowd of diverse people for data processing or analysis purposes.
- Microtask crowdsourcing is commonly executed by means of dedicated Web platforms, where all the published microtasks are publicly presented to workers, and upon selection of the task, users are directed to a different webpage that the task requires.
- Recent work has shed light on the importance of task design choices made with respect to user interface elements; and on how such choices can influence the quality of work produced and satisfaction among workers.
- Although conversational interfaces have been effectively used in numerous applications, the impact of conversational interfaces in microtask crowdsourcing marketplaces has remained unexplored.
- Lowering the entry barrier for workers to participate effectively in crowdsourcing tasks is an important step towards securing the future of crowd work.
- The availability of effective automated text-based conversational interfaces as an alternative to the traditional Web UI, could broaden the pool of available crowd workers by easing their unfamiliarity with the interface elements.

Chatterbox: Introduction

- Conversational agent definition: a software programmed to automatically interpret and respond to requests expressed in natural language, so to mimic the behavior of a human interlocutor.
- Chatbots definition: a class of conversational agents that prevalently use text as an interaction medium.
- While research on chatbot systems dates back to the 1960s, the growing popularity of messaging platforms has sparked new interest both in industry and academia.

Chatterbox: Background and Related Work

Chatterbox: Experimental Design

- The experiments were performed recruiting workers from the F8 microtask crowdsourcing platform.
- Because the goal of the experiment is to understand if text-based conversational agents can enable microtask crowdsourcing, neither quality levels nor qualification were measured.
- 316 distinct workers executed at least one task; 31 workers executed both web and chatbot jobs.

Chatterbox: Evaluation 

- Test-based conversational agents are witnessing widespread adoption as effective tools to automate repetitive tasks, or as an alternative to traditional information seeking interfaces.
- In this paper, researchers provide evidences of their suitability as microtask crowdsourcing platform.
- Researchers highlighted the importance of task-specific interaction design, but also the convenience of advanced text input interfaces currently available in messaging platforms.
- This work provides plenty of inspirations for future research directs, but more research is needed to better understand the peak performance achievable with different task and content types.
- Researchers are also interested in investigating the utility and performance conversational interfaces addressed to requester, both for task creation and monitoring.

Learn Before

Related

Learn After