If even the training error is higher than the desired performance level, adding more data alone will not allow the dev-error curve to reach that desired level. Training error can only stay the same or get worse as more training data is added, and dev error is usually higher than training error.

Google

A standard learning-curve pattern for high avoidable bias occurs when, at the largest training-set size, training error remains far above the desired performance level. If the gap between training error and dev error is small, this indicates small variance despite the large avoidable bias.

Learning Curve Pattern for High Avoidable Bias

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Why More Data Alone Cannot Fix a High-Bias Learning Curve

Identifying High Avoidable Bias on a Learning Curve

Interpreting the Gap Between Training and Dev Error

Gap Indicating _____ in Learning Curves

Matching Learning Curve Features to Diagnostic Insights

Steps for Diagnosing High Avoidable Bias from a Learning Curve

Question:
Describe the "textbook" example of a learning curve that exhibits high avoidable bias. Be sure to explain the relationship between training error, dev error, and desired performance at the largest training set size.

Sample answer:
In a standard learning curve for high avoidable bias, the model's training error stays far above the desired performance level (which is our estimate of the optimal error rate) even at the largest training-set size. This large gap between training error and desired performance indicates that the model does not do well on the training set, meaning it has large avoidable bias. Furthermore, in this pattern, the gap between the training curve and the dev curve is small, which indicates that the model has small variance despite the bias.

Key points:
- Training error remains far above the desired performance level.
- Large gap between training error and desired performance indicates large avoidable bias.
- Model does not perform well on the training set.
- Small gap between training error and dev error.
- Small gap between training and dev error indicates small variance.

Rubric:
A full credit response must clearly identify the two main gaps at the largest training set size: the large gap between training error and desired performance (indicating high bias) and the small gap between training error and dev error (indicating small variance).

Explaining the Pattern of High Avoidable Bias

Case context:
You are building a speech recognition system. You estimate the optimal error rate (desired performance) to be 2%. After plotting the learning curve using all 10,000 hours of available audio data, you observe that your training error plateaus at 18%, and your dev error is 19%.

Question:
Based on the standard textbook learning curve patterns, what specific problem does your speech recognition system have, and what do the respective gaps between the 2%, 18%, and 19% figures indicate?

Sample answer:
The system suffers from high avoidable bias. The large gap of 16% between the desired performance (2%) and the training error (18%) indicates large avoidable bias, meaning the model is not doing well on the training set. Meanwhile, the small gap of 1% between the training error (18%) and the dev error (19%) indicates that the model has small variance.

Key points:
- The model has high avoidable bias.
- The gap between 2% and 18% indicates large avoidable bias.
- The gap between 18% and 19% indicates small variance.
- The model is performing poorly on the training set itself.

Rubric:
The response must correctly diagnose the primary issue as high avoidable bias. It must explain that the large difference between desired performance and training error shows bias, and the small difference between training and dev error shows small variance.

Diagnosing a Speech Recognition Model's Learning Curve

Question:
In a learning curve that shows high avoidable bias, what specific property of the model is indicated by a small gap between the training error and the dev error?

Sample answer:
A small gap between the training error and the dev error indicates that the algorithm has small variance.

Key points:
- It indicates small variance.

Rubric:
Award points if the learner correctly states that the small gap means small variance or low variance.

Learn Before

Related