- Research in microtask crowdsourcing addressed the integration of crowd work platforms with text-messaging and chatbot systems, to either train the machine learning components of the conversational agent, or substitute artificial intelligence for conversation management purposes.
- An early example of the chat-based crowdsourcing system is Guardian, a semi-automated chat system that helps wrapping up Web-ABIs into spoken dialogue systems. 
- The same authors of Guardian proposed Chorus, a system that allowed end-users to chat directly with crowd workers, who would then propose and vote on candidate responses.
- Evorus is an evolution of Chorus, where conversation automation is obtained by adding, learning, and improving automated responses using past information gained from the crowd.
- These systems demonstrated the technical feasibility of application-specific microtask execution through chatbots.

Chatterbox: Crowd-Powered Conversational Systems

- Previous work addressed the problem of ubiquitous and opportunistic microtask crowdsourcing through user interfaces for mobile devices, either in humanitarian or academic setting.
- While the specific work of the researchers addresses a different class of interaction systems (chatbots vs. native or web-based mobile interface), the systems mentioned in this study (including mCrowd, MobileWorks, etc.) share the same ambition and vision for democratization and scaling up of crowd work.
- The results obtained by their analysis of mobile task types and designs features some interesting commonalities and differences with the findings discussed later.

Chatterbox: Mobile Interfaces for Crowdsourcing

- Some researchers noted that microtask marketplaces were often inaccessible to workers in developing countries, and introduced a mobile-based crowdsourcing platform called MobileWorks for OCR tasks, thereby lowering the barriers for participation.
- Other researchers study usability barriers that were prevalent on AMT which prevented workers with little digital literacy skills from participating and completing works, showing that task instructions, user interface, and the workers’ cultural context corresponded to key usability barriers.
- To overcome these barriers and enable better access and participation, authors proposed the use of simplified user interfaces, simplified task instructions, and language localization.
- Complimenting these prior works, the authors of this study proposed that using conventional interfaces that people may be more familiar with as an alternative to standard web interfaces lowers participation barriers.

Chatterbox: Lowering Barriers for Participation in Microtask Crowdsourcing

- Conversational agent definition: a software programmed to automatically interpret and respond to requests expressed in natural language, so to mimic the behavior of a human interlocutor.
- Chatbots definition: a class of conversational agents that prevalently use text as an interaction medium.
- While research on chatbot systems dates back to the 1960s, the growing popularity of messaging platforms has sparked new interest both in industry and academia.

Boston College

Conversational interfaces can facilitate human-computer interactions, but researchers still speculate as to if conversational interfaces can improve worker experience and work quality in crowdsourcing marketplaces. Researchers explore this concept by designing Chatterbox, aimed at gauging the interest and acceptance by crowdworkers for this type of interface. Results suggest that conversational interfaces can be used effectively for crowdsourcing microtasks, resulting in high satisfaction from workers without a negative impact on task execution time or work quality.

Chatterbox: conversational interfaces for microtask crowdsourcing

Mavridis, P., Huang, O., Qiu, S.. Gadiraju, U., & Bozzon, A. (2019) Chatterbox: conversational interfaces for microtask crowdsourcing. Association for Computing Machinery, 243–251. DOI:https://doi-org.ezproxy.lib.uconn.edu/10.1145/3320435.3320439

- The availability and usage of messaging applications (such as Telegram, Facebook Messenger, etc.), as well as their popularity and recent advances in machine learning capabilities, has sparked new interest in conversational interfaces and chatbots.
- The crowing popularity of conversational interfaces has coincided with flourishing crowdsourcing marketplaces.
- Microtask crowdsourcing allows the interaction with a large crowd of diverse people for data processing or analysis purposes.
- Microtask crowdsourcing is commonly executed by means of dedicated Web platforms, where all the published microtasks are publicly presented to workers, and upon selection of the task, users are directed to a different webpage that the task requires.
- Recent work has shed light on the importance of task design choices made with respect to user interface elements; and on how such choices can influence the quality of work produced and satisfaction among workers.
- Although conversational interfaces have been effectively used in numerous applications, the impact of conversational interfaces in microtask crowdsourcing marketplaces has remained unexplored.
- Lowering the entry barrier for workers to participate effectively in crowdsourcing tasks is an important step towards securing the future of crowd work.
- The availability of effective automated text-based conversational interfaces as an alternative to the traditional Web UI, could broaden the pool of available crowd workers by easing their unfamiliarity with the interface elements.

Chatterbox: Introduction

Chatterbox: Background and Related Work

Researchers put together a list of 5 types of microtasks, typically completed by crowd workers in microwork crowdsourcing marketplaces, selected to stress the diversity of evaluated content types and the diversity of UI elements used to perform the tasks.

Chatterbox: Experimental Design

- The experiments were performed recruiting workers from the F8 microtask crowdsourcing platform.
- Because the goal of the experiment is to understand if text-based conversational agents can enable microtask crowdsourcing, neither quality levels nor qualification were measured.
- 316 distinct workers executed at least one task; 31 workers executed both web and chatbot jobs.

Chatterbox: Evaluation 

- Test-based conversational agents are witnessing widespread adoption as effective tools to automate repetitive tasks, or as an alternative to traditional information seeking interfaces.
- In this paper, researchers provide evidences of their suitability as microtask crowdsourcing platform.
- Researchers highlighted the importance of task-specific interaction design, but also the convenience of advanced text input interfaces currently available in messaging platforms.
- This work provides plenty of inspirations for future research directs, but more research is needed to better understand the peak performance achievable with different task and content types.
- Researchers are also interested in investigating the utility and performance conversational interfaces addressed to requester, both for task creation and monitoring.

Learn Before

Related

Learn After