In the experiment, workers iteratively caption the same segment, earning points in proportion to the degree of similarity between the outputs. The gamified app CapCap is utilized for this.

• In the pilot period, 713 video segments with English narrations were distributed to 105 volunteer workers. A total of 60 segments were completed, with an average word error rate (WER) of 4.0%.
• The workers consisted of 15 native English speakers and 81 native Japanese speakers.
• The skill of each worker, which we defined as the average WER of the captions created by the worker, varied widely from 13.2% to 67.2% (among those who played more than 5 rounds).
• Slightly different incentives were needed for skill-focused workers versus task-oriented workers versus the most skilled workers who produced the highest quality results.