Learn Before
When a Smaller Model Is a Reasonable Variance Fix
Question: In what situation is reducing a model’s capacity a sensible way to limit variance, and what is the main downside of that choice compared with using regularization?
Sample answer: This choice makes sense when the training goal includes faster learning or lower compute and memory use. The trade-off is that a smaller model usually has higher bias, so it often delivers weaker classification results than regularization-based methods.
Key points:
- Reducing capacity is most defensible when training speed, compute, or memory savings matter.
- The main cost is greater bias and, as a result, lower classification accuracy than regularization usually gives.
Rubric: The student must say that shrinking the model is appropriate when faster training or lower computational cost is important, and that its main weakness is reduced classification performance caused by increased bias compared with regularization.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
If you want to reduce variance and extra computation is acceptable, which adjustment is usually preferred over shrinking the model?
Reducing a model’s size is usually the first choice for lowering variance when other options are available.
When trying to reduce variance, adding _____ is usually a better first step than simply shrinking the model.
Match each variance-control idea with its best description.
Arrange the steps for handling a model that appears to have high variance.
Why might a practitioner reduce model size even when another approach would better address variance?
Model Capacity and Variance
A smaller model is most useful when _____ training is the main goal, rather than only trying to lower variance.
Match each goal or situation to the technique or consequence most consistent with the guidance on model size and regularization.
Arrange the reasoning used to decide whether shrinking a model is a good response to high variance.
When Should You Shrink a Model Instead of Regularizing It?
Choosing a Variance Reduction Strategy for a Speech Model with Slow Training
When a Smaller Model Is a Reasonable Variance Fix