Case Study

Estimating the Benefit of More Data

Case context: A recommendation team has graphed dev-set error as the number of labeled user sessions in training increases. They can describe the trend for the points they already have, but they are unsure how to estimate what would happen if the training set were expanded further.

Question: What is the main limitation in their analysis, and what additional plot should they create to improve it?

Sample answer: The analysis relies only on dev-set error, which is not easy to extend beyond the data already observed. The team should also plot training error, since comparing training and dev errors helps estimate how performance may change as more data is added.

Key points:

  • The analysis uses only dev-set error.
  • The main problem is that the dev-error trend is hard to project to larger training sizes.
  • The team should add a training-error plot.
  • Training error provides extra information for estimating the value of more data.

Rubric: A good response should identify that dev error alone is insufficient for projection, explain that extrapolating it is difficult, recommend adding training error, and connect that plot to estimating the effect of larger datasets.

0

1

Updated 2026-08-12

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Strategy

Machine Learning Yearning @ DeepLearning.AI