Method: ask 7 researchers to provide realistic feedback to 7 different paragraphs from Study 1. Researchers also need improve the paragraph by completing the 5 microtasks. They then were asked to answer questions based on the NASA TLX tests about mental demand and hard work of the task.
Design implication 3: The cost of providing feedback via a structured communication mechanism is not always correlated to the value it provides

Macalester College

New York University

Two studies designed to quantify the value and cost of each mechanism. Study 1 measures the added value to crowd workers in completing the tasks, and Study 2 measures the cost of each mechanism for requesters.

Methodology (Communicating Context to the Crowd for Complex Writing Tasks)

Salehi, N., Teevan, J., Iqbal, S., Kamar, E., (2017) Communicating Context to the Crowd for Complex Writing Tasks. Stanford University, Microsoft Research. 
http://dx.doi.org/10.1145/2998181.2998332

Communicating Context to the Crowd for Complex Writing Tasks 

1. Method: create 66 mediocre initial paragraphs on a fixed number of topics as starting points. We then applied each of the five mechanisms to the paragraphs to collect additional context on how to make the text closer to the requester’s intentions.
2. Design Implication 1: Structured communication mechanisms with the crowd are most effective when the content quality is low or average. When content quality is high, limited communication can be counter-productive.
3. Different structured communication mechanisms had different effects on the paragraphs; the initial quality was important in determining the effect. 
4. Design Implication 2: Different structured communication mechanisms are effective at different stages. Early on, when content quality is poor, the requester needs to communicate major issues. Later, for average quality content, the different mechanisms have relatively similar added value.


Learn Before

Related