Effect of model expansion versus regularization on overfitting
Case context: You are training a classifier that achieves very low training error but performs poorly on validation data, which indicates high variance. A colleague suggests two separate changes: increasing the network from 3 hidden layers to 7 hidden layers, or adding weight decay to the current model.
Question: Using the bias-variance tradeoff, explain how each change is likely to affect bias and variance. Which option is better for the current problem?
Sample answer: Making the network larger usually lowers bias, but it can also raise variance, so it is not the best fix for a model that is already overfitting. Adding weight decay usually raises bias a little while lowering variance, which makes it the better choice for reducing overfitting.
Key points:
- A larger network generally lowers bias but can increase variance.
- Weight decay generally lowers variance but can increase bias.
- For high variance, weight decay is the better adjustment.
Rubric: The learner must explain that increasing network size reduces bias but can increase variance, and that adding regularization reduces variance but can increase bias. The answer must also identify regularization as the better response to high variance.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Large Models and More Data Can Weaken the Tradeoff
What is a common effect of adding regularization to a learning algorithm?
True or False: Increasing a neural network’s width or depth often lowers bias, but it can also make overfitting more likely.
Why Bias and Variance Trade Off
What usually happens when you make a neural network larger?
Regularization usually reduces both bias and variance in a model.
Some model changes reduce _____ while often increasing variance, and other changes do the opposite.
Match each sign or adjustment to its role in the bias-variance tradeoff.
Arrange the steps a practitioner follows when using training and dev errors to choose an improvement strategy.
A model is underfitting its training data. Which change is most likely to help?
Lowering one source of error in a model always lowers every other source of error as well.
Effect of regularization on bias and variance
Match each bias-variance concept to its description in practice
Order the steps for deciding whether to increase capacity or strengthen regularization when a model is not performing well.
How do extra predictors and regularization affect bias and variance?
Effect of model expansion versus regularization on overfitting
How do bias and variance move when a learning system changes?