Researchers selected a number of common operations along the basic rhetorical dimensions of writing: mechanics (checking for technical errors) and semantics (more in-depth consideration of meaning, such as in rephrasing), and a few other tasks commonly included in crowdsourcing work-flows. These selected operations would then be subjected to evaluation of complexity levels.

Chain Reactions: The Impact of Order on Microtask Chains--Methods, Selecting Operations

- In order to test each hypothesis, it was necessary to identify sets of "easy" and "hard" operations. To identify the complexity of each proposed operation, a between-subject study was conducted recruiting participants from Amazon Mechanical Turk. Researchers analyzed how task complexity compared along the lines of semantic processing (extent to which meaning is processed during a task), time (spent on task), mental demand, interest, meaningfulness, and open-endedness. 
- Participants were randomly assigned an operation and completed it on a given sentence. They then answered subjective questions about the microtask (to assess mental demand, interest, meaningfulness, open-endedness) and multiple choice questions about the objective meaning of the sentence (to assess semantic processing). A baseline condition was included for this assessment. All sentences were held constant. 
- As expected, semantics-based microtasks exhibited greater semantic processing, took longer to complete, presented more mental demand, and were considered more open-ended. There were significant differences found between mechanical and semantic operations as well. Crowd-based microtasks shared similarities with both mechanical and semantic operations (shorter time and lower mental demand, but higher levels of open-endedness, meaning, and interest). High, medium, and low complexity tasks were selected for inclusion in further study based on these results. 

Chain Reactions: The Impact of Order on Microtask Chains--Methods, Determining Complexity

Operations were designed to be performed on sentences, so sentences were gathered from paper abstracts. Paper abstracts are prominent and complex, condensing a larger amount of information into one paragraph. Researchers targeted sentences that were comprehensible standing alone but also required considerable mental processing. Sentences with technical terminology were removed. The Automated Readability Index was used to find those remaining sentences that were "relatively complex," which researchers determined to be around the 75th percentile.

Chain Reactions: The Impact of Order on Microtask Chains--Methods, Selecting Content

- How are performance and experience of a microtask affected by whether preceding microtasks share the same operation or the same content as the current microtask? 
- Participants asked to perform series of 3 microtasks and evaluate their experience on final microtask using measures of time, quality, mental demand, helpfulness, enjoyment. Researchers varied whether preceding tasks had the same operation (same tasks on different sentences) or same content (different tasks on same sentence) as the final task. Also examined whether these effects depended on complexity level of final task. Sentences randomized and final microtask always same (paraphrase for high-complexity (H), duplicateCheck for low-complexity (L)).
- Hypothesize that L tasks would benefit from same-operation chains and H tasks would benefit from same-content chains.

Chain Reactions: The Impact of Order on Microtask Chains--Methods, Testing Continuity Q

- How are performance and experience of a microtask affected by whether previous microtasks are of the same or different complexity as the current microtask?
- Participants asked to complete a series of 3 microtasks on same content and evaluate their experience on final microtask using measures of time, quality, mental demand, helpfulness, enjoyment. The complexity of the final task was either H or L and the chain type was either same-complexity, different-complexity, or a control with no microtasks before the final task (LLL, HHL, L, HHH, LLH, H). Microtask order was randomized for all tasks except the final one (paraphrase for H, duplicateCheck for L). 
- Hypothesize that performance on an L task will not be affected by whether preceding tasks are of same or different complexity. Performance on an H task will be positively affected by same-complexity chains (HHH) and negatively affected by different-complexity chains (LLH).If switching from H to L/L to H imposes an additional cost, then HHL may have lower performance than L only and LLH may have lower performance than H only. 

Chain Reactions: The Impact of Order on Microtask Chains--Methods, Testing Transitions Q

- How are performance and experience of a complex microtask affected when it is preceded by more simple microtasks? Can performing L microtasks help people start an H microtask sooner while simultaneously accomplishing a unit of work?
- Participants asked to complete series of 3 microtasks, the final task always being an H task (paraphrase). The first 2 tasks were either L or M and the content was either the same or different from the final task (LLH_same, MMH_same, LLH_diff, MMH_diff). An H-only control was included as well. Participants were evaluated based on time, quality, mental demand, FirstTypeTime, momentum, and warm-up. 
- Hypothesize that same-content preceding tasks would yield stronger benefits than different-content preceding tasks and M preceding tasks would be more beneficial than L preceding tasks. 

Chain Reactions: The Impact of Order on Microtask Chains--Methods, Testing Ease in Q

- Selecting operations
- Selecting content
- Determining complexity 
- Testing Continuity question
- Testing Transitions question
- Testing Ease in question

Wake Forest University

Cai, C. J., Iqbal, S. T., &amp; Teevan, J. (2016). Chain Reactions: The Impact of Order on Microtask Chains. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/2858036.2858237 

Chain Reactions: The Impact of Order on Microtask Chains

Prior research indicates that task transition is important and affects overall productivity. In this study, researchers aim to examine how we may optimize task transitions through the ordering of existing microtasks to support continuity, help individuals transition between microtasks, and ease individuals into greater processes. Researchers decided to choose writing as their goal task of interest because they reasoned that it is a domain where thoughtful ordering and chaining of microtasks is especially important.
3 research questions:
- Continuity: How are performance and experience of a microtask affected by whether preceding microtasks share the same operation or the same content as the current microtask?
- Transitions: How are performance and experience of a microtask affected by whether previous microtasks are of the same or different complexity as the current microtask?
- Easing in: How are performance and experience of a complex microtask affected when it is preceded by more simple microtasks?

Chain Reactions: The Impact of Order on Microtask Chains--Purpose

Chain Reactions: The Impact of Order on Microtask Chains--Methods

- Results of Continuity study
- Results of Transitions study
- Results of Ease in study

Chain Reactions: The Impact of Order on Microtask Chains--Results

Microtasks have carryover effects on subsequent microtasks in a chain. Properties of tasks (operation, content, complexity) mediate the efficiency, perceived mental workload, and other variables of future tasks. Below is a list of strategies to improve task efficiency based on the results of this study.
- Build momentum using same-operation L tasks
- Transition to H task using L and M tasks on similar content to build momentum
- Avoid switching content during a series of H tasks
- Switch content when transitioning between H and L tasks

Learn Before

Related

Learn After