In the experiment, workers iteratively caption the same segment, earning points in proportion to the degree of similarity between the outputs. The gamified app CapCap is utilized for this. 

- In the pilot period, 713 video segments with English narrations were distributed to 105 volunteer workers. A total of 60 segments were completed, with an average word error rate (WER) of 4.0%.
- The workers consisted of 15 native English speakers and 81 native Japanese speakers. 
- The skill of each worker, which we defined as the average WER of the captions created by the worker, varied widely from 13.2% to 67.2% (among those who played more than 5 rounds).
- Slightly different incentives were needed for skill-focused workers versus task-oriented workers versus the most skilled workers who produced the highest quality results. 


Crowdsourced Video Captioning (Skill Development Framework for Micro-Tasking)

Researchers describe two micro-tasking projects conducted and analyze them to clarify the requirements for our framework. The two projects have similar implications for the skill development of the workers. These implications span many of the issues discussed in existing crowdsourcing research.

Macalester College

This paper proposed a micro-tasking framework that supports skill development of individual workers. Researchers designed the framework based on the review of two micro-tasking projects we have conducted. It consists of four key components, a tutorial generator, a task dispatcher, a feedback visualizer, and a skill assessment engine. The framework will help developers create sustainable micro-tasking systems


Skill Development Framework for Micro-Tasking

Saito, S.; Watanabe, T.; Kobayashi, M.; Takagi, H.; Skill Development Framework for Micro-Tasking. IBM Research – Tokyo, 5-6-52 Toyosu, Koto, Tokyo 135-8511, Japan


In Crowdwork, the quality of the outcomes is controlled by filtering for workers who have higher skills, without considering the development of the individual workers’ skills. However, if micro-tasking systems support skill development, they can produce outcomes of higher-quality and make their use more sustainable, expanding the new labor market.

Problem identification (Skill Development Framework for Micro-Tasking)

Although some studies have proposed frameworks to solve complex tasks via micro-tasking, they paid little attention to skill development. 
- Satzger et al.
- Kittur et al.
- Weld et al. 
- RABJ


Learn Before

Related

Learn After