Google

As training-set size grows, dev and test error should decrease, but training-set error usually increases. With very small training sets, an algorithm can often memorize the examples and get near-zero training error; with larger and more ambiguous or mislabeled training sets, perfectly fitting every example becomes harder.

Training Error Usually Increases with Training Set Size

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

How does training error usually change as the training set grows?

A tiny training set can make training performance look deceptively strong.

As training-set size grows, training error usually _____.

Match each dataset condition to its expected error behavior.

Order the reasoning that explains rising training error.

Question:
In a concise analytical response, explain why training and dev error can move in opposite directions as training-set size grows.

Sample answer:
A very small training set can often be memorized, producing near-zero training error. As more examples are added, fitting every training example becomes harder, especially when some examples are ambiguous or mislabeled, so training error usually rises. At the same time, dev error should decrease as the training set grows.

Key points:
- Small training sets can be memorized.
- Training error usually increases with training-set size.
- Ambiguous or mislabeled examples make perfect fitting harder.
- Dev error should decrease as training-set size grows.

Rubric:
A strong response accurately identifies both error trends and connects rising training error to reduced memorization and the difficulty of fitting larger, potentially ambiguous or mislabeled datasets.

Explain why more training data can raise training error while lowering dev error.

Case context:
A cat classifier has 0% training error on two examples. After the training set grows to 100 and then 10,000 examples, its training accuracy drops slightly while its dev error decreases. Some examples are ambiguous or mislabeled.

Question:
What should the learner conclude about these changes based on the source?

Sample answer:
The changes match the expected pattern. The two-example set was easy to memorize, but fitting all examples became harder as the set grew, particularly because some were ambiguous or mislabeled. Thus, training error increased while dev error decreased.

Key points:
- Zero training error on two examples can result from memorization.
- Training error usually rises as the training set grows.
- Dev error should fall as the training set grows.
- Ambiguous or mislabeled examples make perfect fitting harder.

Rubric:
The response should diagnose both trends, explain the initial memorization, and identify dataset size plus ambiguous or mislabeled examples as reasons that perfect training fit becomes harder.

Diagnose the error trends after a cat classifier receives more training examples.

Question:
Answer in one to three sentences using the source's explanation.

Sample answer:
With only two examples, the learning algorithm can easily memorize both of them. It can therefore classify the training examples perfectly and obtain 0% training error.

Key points:
- The training set is extremely small.
- The algorithm can memorize both examples.
- Memorization can yield 0% training error.

Rubric:
The answer should state that a two-example training set is easy to memorize and connect memorization to zero training error.

Learn Before

Related