A standard learning-curve pattern for high avoidable bias occurs when, at the largest training-set size, training error remains far above the desired performance level. If the gap between training error and dev error is small, this indicates small variance despite the large avoidable bias.

Learning Curve Pattern for High Avoidable Bias

When training-set performance is already close to the optimal error rate, there is little room for bias improvement. If the algorithm is not generalizing well to the dev set, there is ample room for variance improvement.

Learning Curve Pattern for High Variance

A learning curve can indicate both significant bias and significant variance when training error is much higher than the desired performance level and dev error is much higher than training error. In this case, the algorithm needs changes that reduce both bias and variance.

Learning Curve Pattern for Both High Bias and High Variance

Purpose of plotting training and dev error together

Measuring error only at the rightmost point

Extrapolating the _____ curve

Key Concepts of Interpreting Learning Curves

Steps to comprehensively evaluate algorithm performance

Question:
Contrast the insights gained from examining a full learning curve (plotting both training and dev error) versus measuring error only at the rightmost point. Why is the full curve preferred?

Sample answer:
Measuring error at only the rightmost point provides a static snapshot of performance corresponding strictly to using all currently available training data. This gives no insight into trends or data scaling. By plotting the full learning curve with both training and dev error across different dataset sizes, we gain a comprehensive picture of how the algorithm's performance changes as data increases. Viewing both curves on the same plot allows us to more confidently extrapolate the dev error curve to predict whether gathering more data will be beneficial.

Key points:
- The rightmost point only shows performance for all currently available data.
- The full curve gives a comprehensive picture across different training set sizes.
- Plotting both curves together helps in confidently extrapolating the dev error curve.

Rubric:
A strong response should state that the rightmost point only shows performance on maximum available data, while the full curve reveals performance across different sizes. It must mention that plotting both curves aids in extrapolating the dev error curve.

Advantages of the full learning curve over a single point

Case context:
A machine learning team has measured their algorithm's training and dev set error using their entire dataset of 10,000 examples (representing the rightmost point of their data). They are unsure if they should spend a month collecting 5,000 more examples.

Question:
What visual tool should the team construct, and what specific curves should it include to help them make a confident decision about data collection?

Sample answer:
The team should plot a full learning curve showing both the training error curve and the dev error curve across different subset sizes of their data (e.g., 2,000, 4,000, 6,000, 8,000, and 10,000). By examining the full curves together on the same plot rather than just the final point, they can more confidently extrapolate the dev error curve's trajectory to see if more data is likely to lower the error.

Key points:
- Construct a full learning curve across different training set sizes.
- Include both the training error curve and the dev error curve on the same plot.
- Extrapolate the dev error curve visually to inform the decision.

Rubric:
The answer must propose plotting a full learning curve across different dataset sizes, rather than relying on the single rightmost point. It must also specify plotting both training and dev error to allow for dev error extrapolation.

Deciding whether to collect more training data

Question:
What is the primary advantage of examining both the training error curve and the dev error curve on the same plot, as opposed to looking at them separately?

Sample answer:
Examining both curves on the same plot allows us to more confidently extrapolate the dev error curve.

Key points:
- Helps extrapolate the dev error curve.
- Provides more confidence in the extrapolation.

Rubric:
The response should explicitly state that plotting them together aids in the confidence of extrapolating the dev error.

Benefit of observing both error curves

Meaning of the rightmost point

Comprehensive picture from full curves

Plotting both training error and dev error on the same learning curve makes it easier to extrapolate dev-error behavior. Looking at the full curve gives a more comprehensive picture of algorithm performance across different training-set sizes than measuring training and dev error only at the largest available training set size.

Google

A learning curve plots a learning algorithm's dev-set error against the number of training examples.

Learning Curve for Dev-Set Error

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

To plot a learning curve, run the algorithm using different training-set sizes and then plot how dev-set error varies with training-set size. For example, with 1,000 examples, separate copies of the algorithm might be trained on 100, 200, 300, and more examples up to 1,000.

Constructing a Learning Curve by Varying Training Set Size

As training-set size increases, dev-set error should decrease.

Dev-Set Error Should Decrease as Training Set Size Increases

A desired error rate is a level of error one hopes a learning algorithm will eventually achieve. The desired level of performance can be added to a learning curve.

Desired Error Rate for a Learning Algorithm

After adding the desired performance level to a learning curve, visually extrapolating the dev-error curve can help guess how much closer adding more data could get to the desired level. In the passage's example, doubling the training-set size looked plausibly sufficient to reach the desired performance.

Using a Dev-Error Learning Curve to Estimate the Benefit of More Data

If one only looks at the dev-error curve, it can be hard to extrapolate and predict exactly where the curve will go with more data. Training error is an additional plot that can help estimate the impact of adding more data.

Training Error Plot for Estimating the Effect of More Data

Interpreting Learning Curves with Training and Dev Error

When the training set is small, learning-curve points can fluctuate because randomly chosen small subsets may be unusually good or bad, such as containing many ambiguous or mislabeled examples. Skewed class distributions or many-class problems increase the chance of selecting an unrepresentative small subset.

Small Training Sets Can Make Learning Curves Noisy

Identifying the axes of a learning curve

Purpose of a learning curve

A learning curve plots your _____ error against the number of training examples.

Components of a learning curve

Steps to construct a learning curve

Question:
Explain what a learning curve is and why it might be considered an informative technique in machine learning.

Sample answer:
A learning curve is a diagnostic plot that graphs a learning algorithm's dev-set error against the number of training examples used. It is informative because it visually demonstrates how the algorithm's performance on unseen data (the dev set) changes as it is exposed to more training data. This allows practitioners to observe trends and make informed decisions.

Key points:
- Plots dev-set error
- Plots against number of training examples
- Considered highly informative for observing trends

Rubric:
The student must define the axes of a learning curve (dev-set error vs. number of training examples) and explain its informational value.

Analyzing the utility of learning curves

Case context:
You are developing a machine learning model, and its current dev-set error is unacceptably high. You want to determine if investing months into gathering more training data will actually help reduce the error.

Question:
How should you use a learning curve in this situation to inform your decision?

Sample answer:
You should plot a learning curve by calculating and graphing the dev-set error for various subsets of your current training data. By observing the trend of the dev-set error as the number of training examples increases, you can extrapolate whether the curve is still trending downwards (meaning more data will likely help) or if it has flattened out (meaning more data will not significantly reduce the error).

Key points:
- Plot dev-set error against training set size
- Observe the trend as training examples increase
- Extrapolate to make a decision about gathering data

Rubric:
The answer should identify plotting the dev-set error against the number of training examples to observe the trend and evaluate the benefit of more data.

Applying learning curves to diagnose performance

Question:
What two specific variables are plotted against each other in a learning curve?

Sample answer:
A learning curve plots the dev-set error against the number of training examples.

Key points:
- Dev-set error
- Number of training examples

Rubric:
Must mention dev-set error and number of training examples.

Learn Before

Related

Learn After