If even the training error is higher than the desired performance level, adding more data alone will not allow the dev-error curve to reach that desired level. Training error can only stay the same or get worse as more training data is added, and dev error is usually higher than training error.

Why More Data Alone Cannot Fix a High-Bias Learning Curve

Identifying High Avoidable Bias on a Learning Curve

Interpreting the Gap Between Training and Dev Error

Gap Indicating _____ in Learning Curves

Matching Learning Curve Features to Diagnostic Insights

Steps for Diagnosing High Avoidable Bias from a Learning Curve

Question:
Describe the "textbook" example of a learning curve that exhibits high avoidable bias. Be sure to explain the relationship between training error, dev error, and desired performance at the largest training set size.

Sample answer:
In a standard learning curve for high avoidable bias, the model's training error stays far above the desired performance level (which is our estimate of the optimal error rate) even at the largest training-set size. This large gap between training error and desired performance indicates that the model does not do well on the training set, meaning it has large avoidable bias. Furthermore, in this pattern, the gap between the training curve and the dev curve is small, which indicates that the model has small variance despite the bias.

Key points:
- Training error remains far above the desired performance level.
- Large gap between training error and desired performance indicates large avoidable bias.
- Model does not perform well on the training set.
- Small gap between training error and dev error.
- Small gap between training and dev error indicates small variance.

Rubric:
A full credit response must clearly identify the two main gaps at the largest training set size: the large gap between training error and desired performance (indicating high bias) and the small gap between training error and dev error (indicating small variance).

Explaining the Pattern of High Avoidable Bias

Case context:
You are building a speech recognition system. You estimate the optimal error rate (desired performance) to be 2%. After plotting the learning curve using all 10,000 hours of available audio data, you observe that your training error plateaus at 18%, and your dev error is 19%.

Question:
Based on the standard textbook learning curve patterns, what specific problem does your speech recognition system have, and what do the respective gaps between the 2%, 18%, and 19% figures indicate?

Sample answer:
The system suffers from high avoidable bias. The large gap of 16% between the desired performance (2%) and the training error (18%) indicates large avoidable bias, meaning the model is not doing well on the training set. Meanwhile, the small gap of 1% between the training error (18%) and the dev error (19%) indicates that the model has small variance.

Key points:
- The model has high avoidable bias.
- The gap between 2% and 18% indicates large avoidable bias.
- The gap between 18% and 19% indicates small variance.
- The model is performing poorly on the training set itself.

Rubric:
The response must correctly diagnose the primary issue as high avoidable bias. It must explain that the large difference between desired performance and training error shows bias, and the small difference between training and dev error shows small variance.

Diagnosing a Speech Recognition Model's Learning Curve

Question:
In a learning curve that shows high avoidable bias, what specific property of the model is indicated by a small gap between the training error and the dev error?

Sample answer:
A small gap between the training error and the dev error indicates that the algorithm has small variance.

Key points:
- It indicates small variance.

Rubric:
Award points if the learner correctly states that the small gap means small variance or low variance.

Significance of a Small Training-Dev Error Gap

Estimating Optimal Error Rate

Training Set Performance in High Bias Scenarios

A standard learning-curve pattern for high avoidable bias occurs when, at the largest training-set size, training error remains far above the desired performance level. If the gap between training error and dev error is small, this indicates small variance despite the large avoidable bias.

Google

Plotting both training error and dev error on the same learning curve makes it easier to extrapolate dev-error behavior. Looking at the full curve gives a more comprehensive picture of algorithm performance across different training-set sizes than measuring training and dev error only at the largest available training set size.

Interpreting Learning Curves with Training and Dev Error

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Learning Curve Pattern for High Avoidable Bias

When training-set performance is already close to the optimal error rate, there is little room for bias improvement. If the algorithm is not generalizing well to the dev set, there is ample room for variance improvement.

Learning Curve Pattern for High Variance

A learning curve can indicate both significant bias and significant variance when training error is much higher than the desired performance level and dev error is much higher than training error. In this case, the algorithm needs changes that reduce both bias and variance.

Learning Curve Pattern for Both High Bias and High Variance

Purpose of plotting training and dev error together

Measuring error only at the rightmost point

Extrapolating the _____ curve

Key Concepts of Interpreting Learning Curves

Steps to comprehensively evaluate algorithm performance

Question:
Contrast the insights gained from examining a full learning curve (plotting both training and dev error) versus measuring error only at the rightmost point. Why is the full curve preferred?

Sample answer:
Measuring error at only the rightmost point provides a static snapshot of performance corresponding strictly to using all currently available training data. This gives no insight into trends or data scaling. By plotting the full learning curve with both training and dev error across different dataset sizes, we gain a comprehensive picture of how the algorithm's performance changes as data increases. Viewing both curves on the same plot allows us to more confidently extrapolate the dev error curve to predict whether gathering more data will be beneficial.

Key points:
- The rightmost point only shows performance for all currently available data.
- The full curve gives a comprehensive picture across different training set sizes.
- Plotting both curves together helps in confidently extrapolating the dev error curve.

Rubric:
A strong response should state that the rightmost point only shows performance on maximum available data, while the full curve reveals performance across different sizes. It must mention that plotting both curves aids in extrapolating the dev error curve.

Advantages of the full learning curve over a single point

Case context:
A machine learning team has measured their algorithm's training and dev set error using their entire dataset of 10,000 examples (representing the rightmost point of their data). They are unsure if they should spend a month collecting 5,000 more examples.

Question:
What visual tool should the team construct, and what specific curves should it include to help them make a confident decision about data collection?

Sample answer:
The team should plot a full learning curve showing both the training error curve and the dev error curve across different subset sizes of their data (e.g., 2,000, 4,000, 6,000, 8,000, and 10,000). By examining the full curves together on the same plot rather than just the final point, they can more confidently extrapolate the dev error curve's trajectory to see if more data is likely to lower the error.

Key points:
- Construct a full learning curve across different training set sizes.
- Include both the training error curve and the dev error curve on the same plot.
- Extrapolate the dev error curve visually to inform the decision.

Rubric:
The answer must propose plotting a full learning curve across different dataset sizes, rather than relying on the single rightmost point. It must also specify plotting both training and dev error to allow for dev error extrapolation.

Deciding whether to collect more training data

Question:
What is the primary advantage of examining both the training error curve and the dev error curve on the same plot, as opposed to looking at them separately?

Sample answer:
Examining both curves on the same plot allows us to more confidently extrapolate the dev error curve.

Key points:
- Helps extrapolate the dev error curve.
- Provides more confidence in the extrapolation.

Rubric:
The response should explicitly state that plotting them together aids in the confidence of extrapolating the dev error.

Learn Before

Related

Learn After