- The web interface was developed on the FigureEight platform, which provides a standardized way to specify work interfaces in an HTML-like format.
- Researchers chose to use only standard interface elements which are typical of crowdsourcing tasks on FigureEight, to elicit normal interactions of workers with the web interface.
- Figure 1 illustrates the five different task types chosen for this experiment, and how they appear in a standard web interface vs. a conversational interface.

Chatterbox: Standard Web Interface

- Researchers designed and implemented a conversational interface using the Telegram messaging platform, comprised of two main modules: Conversational Management Module and Input Management Module.
- Microtask crowdsourcing user interfaces are typically designed to be minimalistic and easy to use, to enable fast and effective work execution.
- At the beginning of a task execution stage, a chatbot that drives the conversation prompts the worker with messages containing task instructions, including an explanation of the task at hand, and examples of how input could be provided.
- Once no more annotations are pending in the task, the chatbot prompts the next question to the worker and waits for the worker’s response.
- The answer provided then is validated, with positive feedback if acceptable, or a re-submission sequence if the answer is not valid.
- When no more annotations are pending, workers are shown their answers for review and can re-process a previously submitted answer.
- Figure 1 illustrates the five different task types chosen for this experiment, and how they appear in a standard web interface vs. a conversational interface.

Chatterbox: Conversational Interface

- Researchers focused on three types of UI elements that are required to perform the task types investigated in the experiment, including “free text,” “single selection from list,” and “multiple selection from list.”
- Table 1 provides a summary of considered UI elements and their implementation in web and conversational interfaces.

Boston College

Researchers put together a list of 5 types of microtasks, typically completed by crowd workers in microwork crowdsourcing marketplaces, selected to stress the diversity of evaluated content types and the diversity of UI elements used to perform the tasks.

Chatterbox: Experimental Design

Mavridis, P., Huang, O., Qiu, S.. Gadiraju, U., & Bozzon, A. (2019) Chatterbox: conversational interfaces for microtask crowdsourcing. Association for Computing Machinery, 243–251. DOI:https://doi-org.ezproxy.lib.uconn.edu/10.1145/3320435.3320439

Chatterbox: conversational interfaces for microtask crowdsourcing

- Information finding: workers are tasked to find specific relevant information from a given data source, comprised of 50 tasks from 17 business records.
- Human OCR (CAPTCHA): workers are tasked with transcribing the text contained in a CAPTCHA image, comprised of 50 distinct CAPTCHAS of 4 characters.
- Speech transcription: workers are tasked with transcribing recordings of English speech retrieved from Tatoeba, comprised of 50 distinct recordings with length ranging from 2 to 8 seconds.
- Sentiment analysis: workers are tasked with assessing the sentiment of user reviews, comprised of 50 reviews with the overall sentiments reading as positive, negative, or neutral.
- Image annotation: workers are tasked with determining the categories of food items contained in an image, comprised of 50 distinct images, with categories such as eggs, fish, meat, vegetables, fruits, cheese, mushroom, grain, sweets, and non-food.

Chatterbox: Task Types

Chatterbox: Work Interfaces

- To answer RQ1, researchers designed 12 experimental conditions, with working interface type (Web, conversational) and task type as independent variables.
- Instructions at the beginning of the conversational tasks are relatively long, thus possibly affecting task execution time, and to account for this, researchers include 6 additional experimental conditions where the conversational interface has task instructions partially hidden, and workers could instruct the chatbot through specific commands to display more detailed instructions.
- With RQ2, researchers tested 3 custom keyboard configurations with the sentiment analysis and image annotation tasks, thus adding 6 additional experimental conditions.

Chatterbox: Experimental Conditions

On the FigureEight (F8) platform, researchers set up two types of tasks: 
- Web jobs
- Conversational jobs.

Chatterbox: Task Assignment and Execution

- The dependent variables in this experiment include execution time, answer quality, and workers satisfaction.
- Execution time is measured as the time (in seconds) between the start and submission of the task.
- Answer quality is measured by comparing the worker answers with ground truth sentiment analysis and image annotation; information finding and speech transcription tasks were inspected manually; human OCR task answers were compared to the label of the CAPTCHA.
- Worker satisfaction of both web and chatbot tasks is measured by default task ratings on F8 after workers finish the task.

Learn Before

Related

Learn After