Addressing Variance When Adding Features

Impact of New Features on Bias and Variance

Inspiring New Features with _____ Insights

Roles in Feature Modification

Workflow for Feature Modification via Error Analysis

Question:
Explain the theoretical risk of adding more input features based on error analysis, and describe the practical solution recommended to mitigate this risk.

Sample answer:
In theory, adding more input features to an algorithm could increase the model's variance because it increases the complexity of the model. However, in practice, if an increase in variance is actually observed after adding features inspired by error analysis, the recommended solution is to apply regularization. Regularization will usually eliminate this increase in variance, allowing the new features to successfully help with both bias and variance without causing overfitting.

Key points:
- Adding more features theoretically increases variance.
- Regularization is the recommended practical solution.
- Regularization usually eliminates the increase in variance.

Rubric:
A strong response will identify increased variance as the theoretical risk and explicitly name regularization as the practical solution to eliminate that risk.

Theory vs. Practice in Adding Features

Case context:
A machine learning engineer performs error analysis on a speech recognition system and notices it frequently fails to transcribe audio with background traffic noise. To eliminate this particular category of errors, the engineer creates additional input features specifically designed to filter out traffic frequencies. While the avoidable bias decreases significantly, the engineer notices the model's variance has now increased.

Question:
Based on the principles for reducing avoidable bias, what specific technique should the engineer decide to use next to handle the newly introduced variance?

Sample answer:
The engineer should use regularization, which will usually eliminate the increase in variance caused by the addition of the new traffic-filtering features.

Key points:
- The engineer successfully added features based on error analysis to reduce bias.
- The theoretical consequence of increased variance occurred.
- Regularization is the correct technique to eliminate the variance increase.

Rubric:
The answer must correctly identify regularization as the required intervention to fix the variance increase.

Managing Variance After Feature Addition

Question:
When error analysis inspires you to create additional features, what specific goal regarding errors are these new features intended to achieve?

Sample answer:
The additional features are intended to help the algorithm eliminate a particular category of errors identified during the error analysis.

Key points:
- Target specific errors identified in analysis.
- Eliminate a particular category of errors.

Rubric:
Must mention that the goal is to eliminate a "particular category of errors" or target specific types of errors found during analysis.

Target of Additional Features

Benefits of Error-Analysis Inspired Features

Regularization and Increased Variance

Error-analysis insights can motivate creating additional input features that help eliminate a particular category of errors. These new features can help with both bias and variance; if adding features increases variance in practice, regularization will usually eliminate the increase.

Google

When a learning algorithm suffers from high avoidable bias, useful techniques include increasing model size, modifying input features based on error-analysis insights, reducing or eliminating regularization, and modifying the model architecture to better suit the problem. Adding more training data is not a helpful high-bias technique because it helps variance but usually has no significant effect on bias.

Techniques for Reducing Avoidable Bias

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Modifying Input Features Based on Error Analysis to Reduce Avoidable Bias

Reducing or eliminating regularization, such as L2 regularization, L1 regularization, or dropout, can reduce avoidable bias, but it increases variance.

Learn Before

Related

Learn After