Google

If you only inspect the development error curve, it may be difficult to estimate how performance will change as the training set grows. A training-error curve provides another signal that can help assess whether additional data is likely to improve the model.

Using Training Error to Judge the Value of More Data

As the training set grows, development and test error usually move downward, but the error measured on the training set often moves upward. With a very small dataset, a model can often memorize the examples and achieve nearly perfect training performance. When the dataset becomes larger, more varied, or noisier, fitting every example exactly becomes harder, so training error tends to increase.

More Data Often Raises Training Error

A learning system is usually optimized to fit the training examples it sees during learning, so it typically makes fewer mistakes on the training set than on the development set. As a result, the dev-error curve is usually above the training-error curve.

Training Set Error Is Usually Lower Than Dev Set Error

Why plot training error when judging whether more data might improve a model?

A development-error curve computed by itself can be hard to project to much larger training sets.

The extra plot used to judge whether more data will help is _____.

Match each learning-curve element to its use when thinking about the value of additional data.

Arrange the steps for judging whether more data may help a model.

Question:
Write a brief analytical response explaining why the dev-error curve by itself is not enough when judging whether additional data will help, and how the training-error curve contributes.

Sample answer:
The dev-error curve alone can be hard to extend into the future, so it may not tell you reliably what will happen if the training set becomes larger. Adding the training-error curve gives a second perspective that helps estimate how performance might change as more data is collected. For that reason, training error is best used alongside dev error when evaluating the benefit of more data.

Key points:
- The dev-error curve by itself is difficult to extend forward.
- The question is what may happen to dev error after adding more data.
- Training error provides a second curve.
- The second curve helps estimate the effect of more data.

Rubric:
A strong response explains the difficulty of extending the dev-error curve, connects the issue to predicting dev performance with more data, and states that the training-error curve is an additional signal for estimating the effect of extra data.

How does training error help assess the value of more data?

Case context:
A recommendation team has graphed dev-set error as the number of labeled user sessions in training increases. They can describe the trend for the points they already have, but they are unsure how to estimate what would happen if the training set were expanded further.

Question:
What is the main limitation in their analysis, and what additional plot should they create to improve it?

Sample answer:
The analysis relies only on dev-set error, which is not easy to extend beyond the data already observed. The team should also plot training error, since comparing training and dev errors helps estimate how performance may change as more data is added.

Key points:
- The analysis uses only dev-set error.
- The main problem is that the dev-error trend is hard to project to larger training sizes.
- The team should add a training-error plot.
- Training error provides extra information for estimating the value of more data.

Rubric:
A good response should identify that dev error alone is insufficient for projection, explain that extrapolating it is difficult, recommend adding training error, and connect that plot to estimating the effect of larger datasets.

Estimating the Benefit of More Data

Question:
Answer in one to three sentences: What weakness of looking only at the dev-error curve motivates also plotting training error?

Sample answer:
The dev-error curve alone can be hard to project forward, so it does not clearly show what will happen as more data are added. Plotting training error gives another signal that helps estimate how performance changes with additional data.

Key points:
- Dev error by itself may be difficult to project
- Training error adds information about the effect of more data

Rubric:
The answer should mention that the dev-error curve alone is hard to extrapolate and that training error is added to help estimate the impact of more data.

Learn Before

Related