Google

When a learning algorithm suffers from high avoidable bias, useful techniques include increasing model size, modifying input features based on error-analysis insights, reducing or eliminating regularization, and modifying the model architecture to better suit the problem. Adding more training data is not a helpful high-bias technique because it helps variance but usually has no significant effect on bias.

Techniques for Reducing Avoidable Bias

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Error-analysis insights can motivate creating additional input features that help eliminate a particular category of errors. These new features can help with both bias and variance; if adding features increases variance in practice, regularization will usually eliminate the increase.

Modifying Input Features Based on Error Analysis to Reduce Avoidable Bias

Reducing or eliminating regularization, such as L2 regularization, L1 regularization, or dropout, can reduce avoidable bias, but it increases variance.

Reducing or Eliminating Regularization to Reduce Avoidable Bias

Which intervention directly targets high avoidable bias by improving training-set fit?

Adding more training data usually has no significant effect on avoidable bias.

Complete the remedy: Reduce or eliminate _____ to lower avoidable bias.

Match each intervention with its stated effect or rationale.

Order the reasoning process for responding to poor training-set performance.

Question:
In a concise response, explain how increasing model size, modifying input features, reducing regularization, and changing architecture can address high avoidable bias. Contrast them with adding data.

Sample answer:
Increasing model size can reduce bias by allowing the algorithm to fit the training set better. Error analysis can motivate new input features that eliminate particular categories of errors. Reducing or eliminating L1, L2, or dropout regularization lowers avoidable bias but increases variance. A model architecture better suited to the problem can affect both bias and variance. In contrast, adding training data helps variance problems and usually has no significant effect on bias.

Key points:
- Larger models can fit the training set better and reduce bias.
- Feature changes should follow insights from error analysis.
- Reducing regularization lowers avoidable bias but increases variance.
- Architecture changes can affect both bias and variance.
- Additional training data usually has no significant effect on bias.

Rubric:
A strong response accurately explains all four recommended techniques, states the relevant bias-variance effects, and clearly distinguishes additional training data as a variance-oriented intervention rather than a high-bias remedy.

Explain how the four recommended techniques address high avoidable bias.

Case context:
A team finds that its learning algorithm is doing poorly on the training set. Error analysis also reveals a particular recurring category of errors. The team is considering a larger model, new features, weaker regularization, a revised architecture, or more training data.

Question:
What should the team diagnose, which interventions are supported, and which proposed intervention should not be relied upon to solve the diagnosed problem?

Sample answer:
The team should diagnose high avoidable bias. It can increase model size to improve training-set fit, create input features based on the recurring error category, reduce or eliminate regularization while recognizing that variance may increase, or modify the architecture to better suit the problem. It should not rely on more training data to reduce bias because additional data helps variance and usually has no significant effect on bias.

Key points:
- Poor training-set performance indicates high avoidable bias.
- A larger model is a supported intervention.
- Error analysis can motivate useful feature changes.
- Reducing regularization can increase variance.
- Architecture modification may affect both bias and variance.
- More data is not a helpful high-bias technique.

Rubric:
The response should diagnose high avoidable bias, identify the four supported intervention categories, connect feature modification to error analysis, mention the variance tradeoff of reducing regularization, and reject more data as the main bias remedy.

Choose remedies for a model that performs poorly on its training set.

Question:
Answer in one to three sentences: Why might reducing or eliminating L1, L2, or dropout regularization help when avoidable bias is high, and what is the tradeoff?

Sample answer:
Reducing or eliminating regularization can lower avoidable bias. The tradeoff is that variance increases.

Key points:
- Regularization can be reduced or eliminated.
- Avoidable bias decreases.
- Variance increases.

Rubric:
The answer must state both effects: reduced avoidable bias and increased variance.

Learn Before

Related