Google

Very different techniques should be applied depending on whether a project's current problem is high avoidable bias or high variance. Insights from estimating avoidable and unavoidable bias and variance can be used to prioritize techniques that reduce bias versus techniques that reduce variance.

Deciding Whether to Reduce Bias, Variance, or Data Mismatch

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Some methods can simultaneously reduce bias and variance by making major changes to the system architecture, but these methods tend to be harder to identify and implement. Selecting a model architecture that is well suited for the task is one way to reduce both bias and variance simultaneously, although selecting such an architecture can be difficult.

Methods That Simultaneously Reduce Both Bias and Variance

A simple formula for addressing bias and variance is to increase model size when a system has high avoidable bias, and to add data to the training set when a system has high variance.

Simple Bias/Variance Remediation Formula

Different model architectures can have different amounts of bias and variance for a given problem, and modifying the architecture to better suit the problem can affect both bias and variance. Trying new architectures is less predictable than simply increasing model size or adding data.

Modifying Model Architecture Can Affect Both Bias and Variance

What does understanding which component of error is more pressing help you do in an ML project?

The same techniques that reduce bias in a model will also effectively reduce its variance.

Developing good intuition about _____ and Variance will help you choose effective changes for your algorithm.

Match each type of algorithm change or observation to the error component it primarily addresses.

Order the steps in the recommended process for deciding which source of error to address in your ML project.

Which pair of error rates does ML Yearning recommend examining to estimate avoidable bias and variance?

Analyzing which error types your algorithm suffers from most can help you decide whether to focus on reducing data mismatch.

ML Yearning describes using bias/variance analysis to prioritize techniques that reduce bias vs. techniques that reduce _____.

Match each observed error pattern to the remediation focus it suggests.

Order the reasoning steps for deciding whether high bias or high variance is the more pressing problem.

Question:
Explain how a machine learning team can use estimates of avoidable bias and variance to choose and prioritize effective changes for their algorithm. In your explanation, detail how the intended effects of techniques addressing bias differ from the intended effects of techniques addressing variance.

Sample answer:
A machine learning team can use estimates of avoidable bias and variance, obtained by looking at training and dev set error rates, to understand which component of error is more pressing to address. Developing this intuition helps them choose effective changes for their algorithm. The intended effects of techniques addressing bias are to improve the model's performance on the training set. In contrast, the intended effects of techniques addressing variance are to help the model generalize better from the training set to the dev/test sets. By prioritizing techniques that target the larger component of error, the team avoids applying inappropriate techniques to their project's current problem.

Key points:
- Training and dev set error rates are used to estimate avoidable bias and variance.
- This analysis helps identify which component of error is more pressing to address.
- Techniques addressing bias improve training performance, while techniques addressing variance help generalize from training to dev/test.

Rubric:
The response should: 1. Explain that training and dev set error rates are examined to estimate avoidable bias and variance. 2. State that this analysis reveals which error component is more pressing to address. 3. Contrast the intended effect of bias-reducing techniques (improving training set performance) with variance-reducing techniques (improving generalization from training to dev/test sets).

Prioritizing Algorithm Changes Based on Error Components

Case context:
You are analyzing a machine learning model where bias/variance analysis indicates the primary source of error is high avoidable bias. Your team is proposing two categories of changes: Group A changes focus on improving performance on the training set, while Group B changes focus on helping the model generalize better from the training set to the dev/test sets.

Question:
Based on the principles in Machine Learning Yearning, which group of changes (Group A or Group B) should you prioritize, and why?

Sample answer:
You should prioritize Group A changes. High avoidable bias means the model's performance on the training set needs improvement. According to Machine Learning Yearning, changes that address bias improve performance on the training set, whereas changes that address variance help the model generalize from the training set to the dev/test sets. Therefore, prioritizing Group A changes directly targets the primary source of error.

Key points:
- Group A changes address avoidable bias by improving training set performance.
- Group B changes address variance by helping the model generalize from training to dev/test sets.
- Prioritizing techniques should align with the current primary problem (high avoidable bias).

Rubric:
The response should: 1. State that Group A changes should be prioritized. 2. Explain that Group A changes address bias and improve performance on the training set. 3. Explain that Group B changes address variance by helping the model generalize from training to dev/test sets, which is not the primary issue here.

Prioritizing Error Mitigation in a Specialized Image Recognition Project

Question:
According to the source, how does understanding the specific types of error your algorithm suffers from the most help you make decisions regarding the project's focus?

Sample answer:
Understanding which types of error the algorithm suffers from the most helps you decide whether to focus your efforts on reducing bias, reducing variance, or reducing data mismatch.

Key points:
- It helps determine if the focus should be on bias, variance, or data mismatch.
- It guides the team to prioritize the most pressing source of error.

Rubric:
The answer should mention that identifying the main error type positions the practitioner to decide whether to focus on reducing bias, reducing variance, or reducing data mismatch.

Learn Before

Related