Explain why model capacity changes the importance of training-data matching.
Question: In a concise analytical response, explain why a large mismatch between auxiliary and target data is less concerning for a sufficiently large model but more concerning under limited computation.
Sample answer: A sufficiently large and flexible model can learn properties of both the auxiliary and target distributions without the sources seriously competing for representation capacity. When the auxiliary dataset is much larger, however, modeling both sources may require substantially more computation and a much larger network. If those resources are unavailable, matching training data more closely to the dev/test distribution becomes more important. Lowering the auxiliary data's weight is a compromise that preserves its use while reducing the capacity required to perform well across both sources.
Key points:
- A large enough model can represent both data sources.
- Limited capacity makes dev/test distribution matching more important.
- A much larger auxiliary dataset can raise computational requirements.
- Lower auxiliary weighting reduces the need for a massive network while retaining the data.
Rubric: A strong response accurately contrasts sufficient and limited capacity, connects a dominant auxiliary source to increased computational demands, and explains lower auxiliary weighting as the source-supported compromise.
0
1
References
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What should a team with limited model capacity prioritize when auxiliary data greatly outnumbers target data?
Does ample model capacity reduce concern about competition between target and auxiliary data?
With constrained computation, give auxiliary images a much _____ weight.
Match each capacity condition or training choice to its implication.
Order the reasoning for choosing an auxiliary-data weighting strategy.
Explain why model capacity changes the importance of training-data matching.
How should a resource-limited image team use a dominant internet dataset?
Why can down-weighting auxiliary data reduce the required network size?
Which observation most strongly signals that auxiliary-image weighting should be reconsidered?
Must a resource-limited team discard all mismatched auxiliary data?