Essay

How model size affects the importance of training-set alignment

Question: In a short analytical response, explain why a large gap between a support dataset and the real evaluation data matters less for a very capable model, but matters more when training resources are tight.

Sample answer: A high-capacity model can often fit patterns from both datasets without forcing them to compete for the same limited representation. If the support dataset is much bigger, though, learning from both sources may demand more parameters and more training time. When budget or compute is constrained, it becomes more valuable to choose data that better resembles the development and test distribution. Reducing the influence of the support data is one practical tradeoff: it still contributes useful signal, but it lowers the burden on the model to cover two different sources equally well.

Key points:

  • Very capable models can absorb two different data sources.
  • Tight capacity makes closeness to dev/test data more important.
  • A much larger support set can increase training cost.
  • Decreasing the support set's influence can keep the data useful without needing a much bigger network.

Rubric: A strong response contrasts high-capacity and resource-limited settings, explains why a dominant support dataset raises training cost, and describes reduced weighting as a reasonable compromise.

0

1

Updated 2026-08-12

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Strategy

Machine Learning Yearning @ DeepLearning.AI