The maximum number of responses allowed does not have much of an impact on the growth rate for GRI, but it increases the growth at small values for GRII.

Constraints on Cost Forecasting: Maximum Number of Responses

Larger seed sizes makes cost forecasting less accurate.

Constraints on Cost Forecasting: Larger Seed Sizes

Smaller seed sizes result in lower growth rates.

Constraints on Cost Forecasting: Smaller Seed Sizes

Under GRII, cost forecasting briefly becomes inaccurate for high budgets.

Constraints on Cost Forecasting: Brief Inaccuracy

1. The maximum number of responses allowed
2. Larger seed sizes
3. Smaller seed sizes
4. Brief inaccuracy

University of Rochester

Cost forecasting is a technique that allows the crowdsourcer to decide between having the crowd generate more tasks or letting the crowd complete some of the already-generated tasks to more efficiently balance resources in a situation where crowdworkers are both proposing new tasks and completing them.

(Proposed in the paper Efficient crowdsouring of crowd-generated microtasks.)

Cost Forecasting

- GRI: Request a new task when the expected cost of a new task is less than the minimum cost of any existing task.
- GRII: Request a new task when the expected cost of a new task is less than the median cost of the set of already-existing tasks.
- GRII is included in case of outliers that would make the overall minimum an inappropriate choice.

Growth Rules to Test Cost Forecasting (Efficient crowdsourcing of crowd-generated microtasks)

Hotaling, A., &amp; Bagrow, J. P. (2020). Efficient crowdsourcing of crowd-generated microtasks. PLOS ONE, 15(12). https://doi.org/10.1371/journal.pone.0244245 

Efficient crowdsourcing of crowd-generated microtasks

Because the data for both Growth Rules fit a power-law distribution, cost forecasting is a bursty process. This means that algorithm alternates between requesting new tasks in short bursts and receiving responses to existing tasks for quite some time.

Cost Forecasting Can Be Modeled as a Bursty Process

Constraints on Cost Forecasting

1. Efficient algorithms applied to crowdsourcing
2. Higher accuracy for smaller budgets
3. Useful for dynamic costs

Advantages of Cost Forecasting

Cost forecasting allows the crowdsourcer apply efficient algorithms to crowd-generated microtasks, which lets the crowdsourcer get high quality results even when the set of tasks is open-ended or if the budget is cut short unexpectedly.

Advantages of Cost Forecasting: Efficient Algorithms Applied to Crowdsourcing

Cost forecasting can result in higher accuracy for smaller budgets (≤ ≈5%) because it dynamically determines growth rate with past and current state of crowdsourcing.

Learn Before

Related

Learn After