1. Method: create 66 mediocre initial paragraphs on a fixed number of topics as starting points. We then applied each of the five mechanisms to the paragraphs to collect additional context on how to make the text closer to the requester’s intentions.
2. Design Implication 1: Structured communication mechanisms with the crowd are most effective when the content quality is low or average. When content quality is high, limited communication can be counter-productive.
3. Different structured communication mechanisms had different effects on the paragraphs; the initial quality was important in determining the effect. 
4. Design Implication 2: Different structured communication mechanisms are effective at different stages. Early on, when content quality is poor, the requester needs to communicate major issues. Later, for average quality content, the different mechanisms have relatively similar added value.


Study 1 (Communicating Context to the Crowd for Complex Writing Tasks)

Method: ask 7 researchers to provide realistic feedback to 7 different paragraphs from Study 1. Researchers also need improve the paragraph by completing the 5 microtasks. They then were asked to answer questions based on the NASA TLX tests about mental demand and hard work of the task.
Design implication 3: The cost of providing feedback via a structured communication mechanism is not always correlated to the value it provides

Study 2 (Communicating Context to the Crowd for Complex Writing Tasks)

Two studies designed to quantify the value and cost of each mechanism. Study 1 measures the added value to crowd workers in completing the tasks, and Study 2 measures the cost of each mechanism for requesters.

Macalester College

This article explores the notion of structured communication: using structured microtasks to support communication in the domain of complex writing. Different mechanisms are presented and compared in order to find the most effective and less costly ways to communicate with the crowd in the domain of writing. 
Proposal: limiting the requester’s interactions can result in less costly interactions that are just as beneficial


Communicating Context to the Crowd for Complex Writing Tasks 

Salehi, N., Teevan, J., Iqbal, S., Kamar, E., (2017) Communicating Context to the Crowd for Complex Writing Tasks. Stanford University, Microsoft Research. 
http://dx.doi.org/10.1145/2998181.2998332

What mechanisms from structured communication in the writing domain effectively transfer context within a crowd work setting ?

Research Question (Communicating Context to the Crowd for Complex Writing Tasks)

1. General Questions 
2. Specific Questions 
3. Free form Comment and edit 
4. Highlight strength and weaknesses 
5. Main Problem identification

Examined Methods (Communicating Context to the Crowd for Complex Writing Tasks)

Design based on three key findings: 1. Structured communication yields information that can improve the paragraph. 2. The effect size of improvement is dependent on the current state of the paragraph and the communication mechanism. 3. Communication mechanisms have different costs associated with them.

Workflow design (Communicating Context to the Crowd for Complex Writing Tasks)

The cost of communication with the system is relatively low, with an average of 10.6 minute to complete the task, 3.2/5 for mental demand of the task and 2.2/5 for hard work.
All researchers reported that they would be willing to use the system again in the future for writing tasks such as a blog post, the abstract, or introduction of a paper, but not for academic purposes or professional writing. 
They found that the paragraph matched what they would have written themselves in terms of content, but not writing style.

Result (Communicating Context to the Crowd for Complex Writing Tasks)

Neglected the communication cost on the worker’s side. 
A requester’s situations, for example, limitations of the mobile device, can affect communication cost.
Future study can be done to examine how to adapt design implication to other domains and other communication mechanisms.

Learn Before

Related

Learn After