- The dependent variables in this experiment include execution time, answer quality, and workers satisfaction.
- Execution time is measured as the time (in seconds) between the start and submission of the task.
- Answer quality is measured by comparing the worker answers with ground truth sentiment analysis and image annotation; information finding and speech transcription tasks were inspected manually; human OCR task answers were compared to the label of the CAPTCHA.
- Worker satisfaction of both web and chatbot tasks is measured by default task ratings on F8 after workers finish the task.

Boston College

Researchers put together a list of 5 types of microtasks, typically completed by crowd workers in microwork crowdsourcing marketplaces, selected to stress the diversity of evaluated content types and the diversity of UI elements used to perform the tasks.

Chatterbox: Experimental Design

Mavridis, P., Huang, O., Qiu, S.. Gadiraju, U., & Bozzon, A. (2019) Chatterbox: conversational interfaces for microtask crowdsourcing. Association for Computing Machinery, 243–251. DOI:https://doi-org.ezproxy.lib.uconn.edu/10.1145/3320435.3320439

Chatterbox: conversational interfaces for microtask crowdsourcing

- Information finding: workers are tasked to find specific relevant information from a given data source, comprised of 50 tasks from 17 business records.
- Human OCR (CAPTCHA): workers are tasked with transcribing the text contained in a CAPTCHA image, comprised of 50 distinct CAPTCHAS of 4 characters.
- Speech transcription: workers are tasked with transcribing recordings of English speech retrieved from Tatoeba, comprised of 50 distinct recordings with length ranging from 2 to 8 seconds.
- Sentiment analysis: workers are tasked with assessing the sentiment of user reviews, comprised of 50 reviews with the overall sentiments reading as positive, negative, or neutral.
- Image annotation: workers are tasked with determining the categories of food items contained in an image, comprised of 50 distinct images, with categories such as eggs, fish, meat, vegetables, fruits, cheese, mushroom, grain, sweets, and non-food.

Chatterbox: Task Types

- Researchers focused on three types of UI elements that are required to perform the task types investigated in the experiment, including “free text,” “single selection from list,” and “multiple selection from list.”
- Table 1 provides a summary of considered UI elements and their implementation in web and conversational interfaces.

Chatterbox: Work Interfaces

- To answer RQ1, researchers designed 12 experimental conditions, with working interface type (Web, conversational) and task type as independent variables.
- Instructions at the beginning of the conversational tasks are relatively long, thus possibly affecting task execution time, and to account for this, researchers include 6 additional experimental conditions where the conversational interface has task instructions partially hidden, and workers could instruct the chatbot through specific commands to display more detailed instructions.
- With RQ2, researchers tested 3 custom keyboard configurations with the sentiment analysis and image annotation tasks, thus adding 6 additional experimental conditions.

Chatterbox: Experimental Conditions

On the FigureEight (F8) platform, researchers set up two types of tasks: 
- Web jobs
- Conversational jobs.

Learn Before

Related