Learn Before
How model size affects the importance of training-set alignment
Question: In a short analytical response, explain why a large gap between a support dataset and the real evaluation data matters less for a very capable model, but matters more when training resources are tight.
Sample answer: A high-capacity model can often fit patterns from both datasets without forcing them to compete for the same limited representation. If the support dataset is much bigger, though, learning from both sources may demand more parameters and more training time. When budget or compute is constrained, it becomes more valuable to choose data that better resembles the development and test distribution. Reducing the influence of the support data is one practical tradeoff: it still contributes useful signal, but it lowers the burden on the model to cover two different sources equally well.
Key points:
- Very capable models can absorb two different data sources.
- Tight capacity makes closeness to dev/test data more important.
- A much larger support set can increase training cost.
- Decreasing the support set's influence can keep the data useful without needing a much bigger network.
Rubric: A strong response contrasts high-capacity and resource-limited settings, explains why a dominant support dataset raises training cost, and describes reduced weighting as a reasonable compromise.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
When a small model is trained with a tiny target dataset and a much larger auxiliary dataset, what should the team focus on first?
A model with enough capacity can often learn patterns from both a main dataset and a smaller auxiliary dataset without necessarily running out of room.
With limited compute, assign extra context a much _____ weight.
Match Each Model Setting to Its Likely Effect
Order the reasoning for selecting a weighting strategy for extra training data.
How model size affects the importance of training-set alignment
How should a small vision team use a much larger auxiliary image collection?
Why does reducing the influence of extra training data ease model-size demands?
Which situation most strongly suggests lowering the weight of a large auxiliary data source?
Should a small team always ignore extra training data that comes from a different distribution?