Effect of Lower Weight on Model Capacity
Question: How does giving a very large auxiliary dataset from a different distribution a smaller training weight affect the size of the neural network you need?
Sample answer: Using a lower weight for the auxiliary data usually means the network does not need to be as large. The model is not being pushed as hard to fit both datasets equally, so the training burden and compute demand are reduced.
Key points:
- A smaller neural network may be sufficient.
- The overall training burden and compute demand are reduced.
Rubric: Acceptable answers should state that the neural network can be smaller, or that the computational burden is reduced.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Choosing the Weight for Auxiliary Data
When is it sensible to give auxiliary training examples less influence during training?
Reducing the weight of secondary data can make model training less demanding when that data is not the main target distribution.
Use re-weighting only when the extra data comes from a _____ distribution than the dev/test set.
When to Reduce the Weight of Auxiliary Data
Order the steps for deciding whether to reduce the weight of auxiliary training data.
Why Equal Weighting Can Be Expensive
True or False: Reducing the weight of auxiliary training data is the same as deleting those examples from training.
When mixing a small auxiliary dataset with a main dataset, reducing its _____ can keep the model balanced.
Match each term to its best description when auxiliary data is used for training.
Order the reasoning that shows why reducing the influence of a large auxiliary dataset can make a compact network sufficient.
When to Reduce the Weight of Auxiliary Data
Weighting Target and Auxiliary Data Under Compute Limits
Effect of Lower Weight on Model Capacity