Cost forecasting allows the crowdsourcer apply efficient algorithms to crowd-generated microtasks, which lets the crowdsourcer get high quality results even when the set of tasks is open-ended or if the budget is cut short unexpectedly.

Advantages of Cost Forecasting: Efficient Algorithms Applied to Crowdsourcing

Cost forecasting can result in higher accuracy for smaller budgets (≤ ≈5%) because it dynamically determines growth rate with past and current state of crowdsourcing.

Advantages of Cost Forecasting: Higher Accuracy for Smaller Budgets

Even though cost forecasting assumes each task is equally costly, it can still be helpful when that is not the case.

Advantages of Cost Forecasting: Useful for Dynamic Costs

1. Efficient algorithms applied to crowdsourcing
2. Higher accuracy for smaller budgets
3. Useful for dynamic costs

University of Rochester

Cost forecasting is a technique that allows the crowdsourcer to decide between having the crowd generate more tasks or letting the crowd complete some of the already-generated tasks to more efficiently balance resources in a situation where crowdworkers are both proposing new tasks and completing them.

(Proposed in the paper Efficient crowdsouring of crowd-generated microtasks.)

Cost Forecasting

Hotaling, A., &amp; Bagrow, J. P. (2020). Efficient crowdsourcing of crowd-generated microtasks. PLOS ONE, 15(12). https://doi.org/10.1371/journal.pone.0244245 

Efficient crowdsourcing of crowd-generated microtasks

- GRI: Request a new task when the expected cost of a new task is less than the minimum cost of any existing task.
- GRII: Request a new task when the expected cost of a new task is less than the median cost of the set of already-existing tasks.
- GRII is included in case of outliers that would make the overall minimum an inappropriate choice.

Growth Rules to Test Cost Forecasting (Efficient crowdsourcing of crowd-generated microtasks)

Because the data for both Growth Rules fit a power-law distribution, cost forecasting is a bursty process. This means that algorithm alternates between requesting new tasks in short bursts and receiving responses to existing tasks for quite some time.

Cost Forecasting Can Be Modeled as a Bursty Process

1. The maximum number of responses allowed
2. Larger seed sizes
3. Smaller seed sizes
4. Brief inaccuracy

Learn Before

Related

Learn After