Google

When additional data is much more numerous than target-distribution data, limited computational resources can make it expensive to model both sources well. Down-weighting the auxiliary source can reduce the need for a very large neural network while still using the additional data.

Training Data Matching Matters More When Model Capacity Is Limited

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

What should a team with limited model capacity prioritize when auxiliary data greatly outnumbers target data?

Does ample model capacity reduce concern about competition between target and auxiliary data?

With constrained computation, give auxiliary images a much _____ weight.

Match each capacity condition or training choice to its implication.

Order the reasoning for choosing an auxiliary-data weighting strategy.

Question:
In a concise analytical response, explain why a large mismatch between auxiliary and target data is less concerning for a sufficiently large model but more concerning under limited computation.

Sample answer:
A sufficiently large and flexible model can learn properties of both the auxiliary and target distributions without the sources seriously competing for representation capacity. When the auxiliary dataset is much larger, however, modeling both sources may require substantially more computation and a much larger network. If those resources are unavailable, matching training data more closely to the dev/test distribution becomes more important. Lowering the auxiliary data's weight is a compromise that preserves its use while reducing the capacity required to perform well across both sources.

Key points:
- A large enough model can represent both data sources.
- Limited capacity makes dev/test distribution matching more important.
- A much larger auxiliary dataset can raise computational requirements.
- Lower auxiliary weighting reduces the need for a massive network while retaining the data.

Rubric:
A strong response accurately contrasts sufficient and limited capacity, connects a dominant auxiliary source to increased computational demands, and explains lower auxiliary weighting as the source-supported compromise.

Explain why model capacity changes the importance of training-data matching.

Case context:
A team evaluates on mobile app images. It has 5,000 mobile app images and 40 times as many internet images, but it cannot afford a neural network large enough to model both sources equally well.

Question:
Diagnose the team's capacity problem and decide how it should weight the internet images according to the source.

Sample answer:
The much larger internet dataset may demand far more computation and representation capacity if the model must perform well on both sources. Because the team cannot build a sufficiently large network and its evaluation distribution is mobile app images, it should pay closer attention to matching that target distribution. It can still use the internet images, but should give them a much lower weight as a compromise.

Key points:
- Internet images greatly outnumber target-distribution images.
- Modeling both sources can exceed available computation or capacity.
- The mobile app distribution should receive greater attention because it matches evaluation.
- Internet images should remain in training but receive a much lower weight.

Rubric:
The response should identify competition for limited capacity, prioritize the mobile app dev/test distribution, and recommend lower—not zero or higher—weight for the internet images.

How should a resource-limited image team use a dominant internet dataset?

Question:
Answer in one to three sentences: Why does assigning a lower weight to abundant auxiliary images reduce the need for a massive neural network?

Sample answer:
A lower weight reduces the pressure on the model to perform equally well on the auxiliary and target sources. The model therefore needs less capacity to represent both while still benefiting from the additional images.

Key points:
- Lower weighting reduces competition for limited representation capacity.
- The model need not be as massive to handle both sources.
- The auxiliary data is retained rather than discarded.

Rubric:
The answer must connect lower auxiliary weight to reduced pressure to model both sources and note that the auxiliary data can still be used.

Learn Before

Related