Google

A learning curve is a plot of a model's validation error versus the number of training examples used to fit it.

Validation-Error Learning Curve

To build this kind of curve, train separate copies of the algorithm on training subsets of increasing size, then plot dev-set error against the number of training examples used. For instance, if 1,000 labeled examples are available, you might train one model on 100 examples, another on 200, another on 300, and continue with larger subsets up to the full 1,000.

Constructing a Performance Curve from Different Training Set Sizes

As the amount of training data increases, dev-set error is expected to decrease.

Larger Training Sets Usually Lower Dev-Set Error

A target error rate is the level of performance you want a model to reach in the future. On a learning curve, it is shown as a reference level so you can compare current errors against the goal.

Target Error Rate on a Learning Curve

After marking the target performance level on a development-set error curve, extending the trend by eye can provide a rough estimate of whether a larger training set might close the gap. In the example, the projected shape suggested that doubling the data could plausibly bring performance to the target.

Using a Development-Set Curve to Estimate the Payoff of More Data

If you only inspect the development error curve, it may be difficult to estimate how performance will change as the training set grows. A training-error curve provides another signal that can help assess whether additional data is likely to improve the model.

Using Training Error to Judge the Value of More Data

Showing training error and development error on the same graph makes it easier to estimate how development error may change as the training set grows. Looking at the full curve also reveals patterns such as high bias, high variance, or both more clearly than checking only the final point at the largest training-set size.

Why plot training and development error together on a learning curve?

When only a few training examples are used, each learning-curve point can vary a lot because the random subset may be unusually easy or unusually hard. A small sample might also overinclude noisy labels, borderline cases, or other awkward examples. This effect is more likely when the classes are highly imbalanced or when the task has many classes, since a tiny draw is then less likely to reflect the full data set.

Tiny Training Samples Make Learning Curves Unstable

What are the axes of a learning curve?

A learning curve shows error versus network depth.

A learning curve shows how your _____ error changes as the training set grows.

Learning Curves and Validation Error

Parts of a learning curve

Constructing a learning curve for model selection

Question:
Describe what a learning curve is in machine learning and explain why it is useful for model diagnosis.

Sample answer:
A learning curve is a graph that shows how a model’s validation or development error changes as the number of training examples increases. One axis represents the amount of training data, and the other axis represents error on held-out data. It is useful because it helps practitioners see whether extra data is still improving the model, whether progress has started to flatten out, and whether the model’s behavior suggests a data, bias, or variance problem.

Key points:
- Horizontal axis: number of training examples
- Vertical axis: dev-set or validation error
- Useful for judging how performance changes as training data grows

Rubric:
The student must identify the two axes of a learning curve and explain how the graph helps diagnose model behavior.

What a learning curve shows

Case context:
You are building a fraud detection model for an online marketplace, and its development-set error is still too high. You are considering whether collecting several more months of labeled transaction data is worth the time and cost.

Question:
How can a learning curve help you decide whether more training data is likely to improve performance?

Sample answer:
Plot the development-set error for several training-set sizes taken from your current data. Then examine how the error changes as the training set gets larger. If the curve is still dropping, the model may benefit from additional labeled examples. If the curve has become nearly flat, more data is unlikely to produce a large improvement.

Key points:
- Plot development-set error against training-set size
- Check how the error changes as more examples are added
- Use the trend to decide whether gathering more data is worthwhile

Rubric:
The answer should describe plotting development-set error versus the number of training examples, using the trend to judge whether additional data is likely to help.

Using a learning curve to judge whether more data will help

Question:
Which two quantities are compared on a learning curve?

Sample answer:
It compares error on the development set with the number of training examples used.

Key points:
- Error measured on the development set
- Number of training examples used

Rubric:
The response must mention both the development-set error and the number of training examples.

Learn Before

Related