Google

When a learning algorithm suffers from high variance, useful techniques include adding more training data, adding regularization, adding early stopping, using feature selection, and sometimes modifying input features or model architecture. Decreasing model size can reduce variance, but should be used with caution because regularization usually gives better classification performance when computational cost is not the concern.

Techniques for Reducing Variance

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Adding regularization, such as L2 regularization, L1 regularization, or dropout, reduces variance but increases bias.

Adding Regularization to Reduce Variance

Early stopping means stopping gradient descent early based on dev-set error. It reduces variance but increases bias, behaves much like regularization, and is sometimes called a regularization technique.

Early Stopping to Reduce Variance

Feature selection can reduce variance by decreasing the number or type of input features, but it can also increase bias. Small reductions in feature count are unlikely to have a large bias effect, while large reductions are more likely to matter, especially if too many useful features are excluded. In modern deep learning with plentiful data, practitioners are more likely to provide all features and let the algorithm learn which to use, but feature selection can still be useful when the training set is small.

Feature Selection for Variance Reduction

Decreasing model size, such as reducing neurons or layers, can decrease variance while increasing bias, but it is not generally recommended as a variance remedy. If computational cost is not the concern, adding regularization usually gives better classification performance; the main advantage of a smaller model is faster training.

Decreasing Model Size as a Variance Remedy

Which technique is described as the simplest and most reliable way to address high variance?

Adding regularization can reduce variance while increasing bias.

Stopping gradient descent early based on _____ error can reduce variance.

Match each variance-reduction technique with its defining consideration.

Order the reasoning process for choosing a remedy for a high-variance algorithm.

Question:
Write a concise analytical response comparing decreasing model size with adding regularization when computational cost is not the concern.

Sample answer:
Decreasing the number of neurons or layers can reduce variance, but it should be used cautiously and is not the recommended remedy. When computational cost is not the concern, adding regularization is generally preferable because it usually gives better classification performance. Regularization reduces variance, although it can increase bias.

Key points:
- Decreasing model size can reduce variance.
- The source recommends using model-size reduction with caution.
- Regularization usually gives better classification performance.
- Regularization is preferred when computational cost is not the concern.
- Regularization can reduce variance while increasing bias.

Rubric:
A strong response explains that a smaller model can reduce variance, notes the source's caution, identifies regularization as the preferred option when computational cost is not a concern, and mentions the variance-bias tradeoff.

Explain why decreasing model size should be used cautiously as a remedy for high variance.

Case context:
A learning algorithm has high variance. The team is considering removing layers to make the model smaller, but computational cost is not a concern.

Question:
What should the team consider doing instead, and why?

Sample answer:
The team should consider adding regularization instead of decreasing the model size. Although a smaller model can reduce variance, the source advises caution and states that regularization usually gives better classification performance when computational cost is not the concern. The team should also recognize that regularization may increase bias.

Key points:
- Recommend adding regularization.
- Decreasing model size should be used cautiously.
- Computational cost is not the concern.
- Regularization usually gives better classification performance.
- Regularization reduces variance but can increase bias.

Rubric:
Full credit requires recommending regularization, explaining its classification-performance advantage over reducing model size in this situation, and acknowledging its possible increase in bias.

Choose a variance remedy when model computation is not a constraint.

Question:
Answer in one to three sentences using the source's description of feature modification.

Sample answer:
Error analysis can reveal a particular category of errors and inspire the creation of additional input features. Those features can help the algorithm eliminate that category of errors.

Key points:
- Use insights from error analysis.
- Create additional input features.
- Target a particular category of errors.

Rubric:
The answer should connect insights from error analysis to creating additional features that address a particular category of errors.

Learn Before

Related