Identifying High Variance from Learning Curves

Room for Bias Improvement

Adding _____ to Close the Error Gap

Matching Learning Curve Concepts

Analyzing a High Variance Learning Curve

Question:
Describe a scenario where an algorithm achieves a training error close to the optimal error rate but has a much higher dev error. Explain what this implies about bias and variance, and discuss the potential for improvement in both areas.

Sample answer:
In this scenario, the algorithm does well on the training set, meaning the training error is close to the optimal error rate. This indicates that the bias is small. However, because the dev error is much higher, the algorithm is not generalizing well to previously unseen data, which means the variance is large. Consequently, there is little room for bias improvement, but there is ample room for variance improvement. Adding more training data is a strategy that would probably help close the gap.

Key points:
- Training error near the optimal rate indicates small bias.
- High dev error relative to training error indicates large variance.
- There is little room for bias improvement.
- There is ample room for variance improvement.
- Adding training data can help close the generalization gap.

Rubric:
A strong response will correctly identify low bias and high variance based on the described errors, and explain that improvement efforts should focus on variance (e.g., adding more data) rather than bias.

Assessing Error Improvement Potential

Case context:
You are reviewing a learning curve for a model where the optimal error rate is known to be around 14%. Your algorithm's training error is currently at 14.5%, but its dev error is at 26%.

Question:
Based on this learning curve, what should you diagnose about the model's bias and variance, and where is the room for improvement?

Sample answer:
The model's training error of 14.5% is very close to the optimal error rate of 14%, indicating that the bias is small and there is not much room for improvement in terms of bias. However, the dev error of 26% is much higher than the training error, showing the model is not generalizing well. This indicates high variance, meaning there is ample room for improvement in errors due to variance, potentially by adding more training data.

Key points:
- Training error near optimal implies small bias.
- A large gap between dev error and training error implies large variance.
- There is little room to improve bias.
- There is ample room to improve variance.

Rubric:
The learner must diagnose low bias with little room for bias improvement, and high variance with ample room for variance improvement.

Diagnosing a Dev Error Gap

Question:
If an algorithm's training error is close to the optimal error rate, why is there more room for variance improvement than bias improvement?

Sample answer:
Because the training performance is already near the optimal rate, the bias is small, leaving little room for bias improvement. If the algorithm fails to generalize, the dev error will be much higher, meaning the variance is large and there is ample room to reduce it.

Key points:
- Training error near the optimal rate means small bias.
- Poor generalization means large variance.
- Ample room exists for variance improvement.

Rubric:
The answer must state that low training error leaves little room for bias reduction, while poor generalization leaves ample room to reduce variance.

Variance vs. Bias Improvement Potential

Effect of Adding Training Data

Generalization and High Variance

When training-set performance is already close to the optimal error rate, there is little room for bias improvement. If the algorithm is not generalizing well to the dev set, there is ample room for variance improvement.

Google

Plotting both training error and dev error on the same learning curve makes it easier to extrapolate dev-error behavior. Looking at the full curve gives a more comprehensive picture of algorithm performance across different training-set sizes than measuring training and dev error only at the largest available training set size.

Interpreting Learning Curves with Training and Dev Error

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

A standard learning-curve pattern for high avoidable bias occurs when, at the largest training-set size, training error remains far above the desired performance level. If the gap between training error and dev error is small, this indicates small variance despite the large avoidable bias.

Learning Curve Pattern for High Avoidable Bias

Learning Curve Pattern for High Variance

A learning curve can indicate both significant bias and significant variance when training error is much higher than the desired performance level and dev error is much higher than training error. In this case, the algorithm needs changes that reduce both bias and variance.

Learning Curve Pattern for Both High Bias and High Variance

Purpose of plotting training and dev error together

Measuring error only at the rightmost point

Extrapolating the _____ curve

Key Concepts of Interpreting Learning Curves

Steps to comprehensively evaluate algorithm performance

Question:
Contrast the insights gained from examining a full learning curve (plotting both training and dev error) versus measuring error only at the rightmost point. Why is the full curve preferred?

Sample answer:
Measuring error at only the rightmost point provides a static snapshot of performance corresponding strictly to using all currently available training data. This gives no insight into trends or data scaling. By plotting the full learning curve with both training and dev error across different dataset sizes, we gain a comprehensive picture of how the algorithm's performance changes as data increases. Viewing both curves on the same plot allows us to more confidently extrapolate the dev error curve to predict whether gathering more data will be beneficial.

Key points:
- The rightmost point only shows performance for all currently available data.
- The full curve gives a comprehensive picture across different training set sizes.
- Plotting both curves together helps in confidently extrapolating the dev error curve.

Rubric:
A strong response should state that the rightmost point only shows performance on maximum available data, while the full curve reveals performance across different sizes. It must mention that plotting both curves aids in extrapolating the dev error curve.

Advantages of the full learning curve over a single point

Case context:
A machine learning team has measured their algorithm's training and dev set error using their entire dataset of 10,000 examples (representing the rightmost point of their data). They are unsure if they should spend a month collecting 5,000 more examples.

Question:
What visual tool should the team construct, and what specific curves should it include to help them make a confident decision about data collection?

Sample answer:
The team should plot a full learning curve showing both the training error curve and the dev error curve across different subset sizes of their data (e.g., 2,000, 4,000, 6,000, 8,000, and 10,000). By examining the full curves together on the same plot rather than just the final point, they can more confidently extrapolate the dev error curve's trajectory to see if more data is likely to lower the error.

Key points:
- Construct a full learning curve across different training set sizes.
- Include both the training error curve and the dev error curve on the same plot.
- Extrapolate the dev error curve visually to inform the decision.

Rubric:
The answer must propose plotting a full learning curve across different dataset sizes, rather than relying on the single rightmost point. It must also specify plotting both training and dev error to allow for dev error extrapolation.

Deciding whether to collect more training data

Question:
What is the primary advantage of examining both the training error curve and the dev error curve on the same plot, as opposed to looking at them separately?

Sample answer:
Examining both curves on the same plot allows us to more confidently extrapolate the dev error curve.

Key points:
- Helps extrapolate the dev error curve.
- Provides more confidence in the extrapolation.

Rubric:
The response should explicitly state that plotting them together aids in the confidence of extrapolating the dev error.

Learn Before

Related

Learn After