Case context: A data scientist has collected 800 labeled examples for a prediction task. To check whether additional data would likely improve performance, the data scientist wants to create a learning curve that shows dev-set error as the training set grows.

Question: Using the idea of a learning curve, explain how the training runs should be organized and what should be plotted.

Sample answer: The data scientist should choose several training-set sizes that increase toward the full 800-example dataset, such as 80, 160, 240, and so on. For each size, a separate copy of the model should be trained on a different subset of that size. After each model is trained, its dev-set error should be measured. The learning curve is then formed by plotting dev-set error on the y-axis against training-set size on the x-axis.

Key points:

• Use multiple training-subset sizes that grow up to the full available set, for example 80, 160, 240, ..., 800.
• Train a separate model for each chosen subset size.
• Measure dev-set error for each trained model.
• Plot dev-set error versus training-set size.

Rubric: The response must explain: 1) choosing increasing subset sizes up to the maximum available, 2) training separate model copies on each subset, and 3) evaluating and plotting dev-set error against the training-set sizes.