Google

When the training set is small, learning-curve points can fluctuate because randomly chosen small subsets may be unusually good or bad, such as containing many ambiguous or mislabeled examples. Skewed class distributions or many-class problems increase the chance of selecting an unrepresentative small subset.

Small Training Sets Can Make Learning Curves Noisy

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

If learning-curve noise makes the true trend hard to see, one remedy is to sample several different training sets of the same small size, train a different model on each, compute each model's training and dev error, and plot the average training and dev error.

Averaging Learning Curves over Multiple Random Subsets

When training data is skewed toward one class or has many classes, learning curves can be less noisy if each small subset is balanced so that class fractions are as close as possible to the overall fractions in the original training set.

Balanced Subsets for Noisy Learning Curves in Skewed or Many-Class Data

Why can a learning-curve point fluctuate when it is based on a very small random training subset?

A small random subset can produce a learning-curve value that is higher or lower than expected.

A small subset with many ambiguous or mislabeled examples is unusually _____.

Match each small-subset condition to its learning-curve implication.

Order the reasoning used to diagnose a noisy point at a small training-set size.

Question:
In a concise analytical response, explain how random subset composition, ambiguous or mislabeled examples, class skew, and many-class data contribute to noisy learning-curve points.

Sample answer:
Each point at a small training-set size may be based on a random subset that does not represent the full dataset well. An unlucky subset can contain many ambiguous or mislabeled examples and produce worse-than-expected error, while a lucky subset can produce better-than-expected error. Heavy class skew and a huge number of classes further increase the chance that a small subset omits or underrepresents important classes. Training and dev errors may therefore fluctuate at the smaller training-set sizes.

Key points:
- Small random subsets may be unrepresentative
- Ambiguous or mislabeled examples can create a particularly bad subset
- A lucky subset can yield better-than-expected values
- Class skew increases the risk of poor class representation
- A huge number of classes increases the risk of an unrepresentative subset

Rubric:
A strong response clearly links small random subsets to error fluctuation, contrasts unusually good and bad subsets, and explains why skewed or many-class data increases unrepresentativeness. Award full credit only when the explanation remains grounded in the supplied evidence.

Explain why small training subsets can make learning curves noisy.

Case context:
A binary classification dataset contains many more negative than positive examples. A learning curve built from increasing random subset sizes shows an unexpected fluctuation at 10 examples. Inspection reveals that the 10-example subset contains only negative examples.

Question:
What should the learner diagnose as the likely reason for the fluctuation, and why would this subset make meaningful learning difficult?

Sample answer:
The learner should diagnose the 10-example subset as unrepresentative because the skewed class distribution allowed the random sample to contain only negative examples. With no positive examples represented, the algorithm has little opportunity to learn something meaningful about distinguishing the two classes, so the corresponding training and dev errors may fluctuate unexpectedly.

Key points:
- The subset is unrepresentative
- Class skew makes such a subset more likely
- The subset contains no positive examples
- Missing positive examples makes meaningful learning difficult
- The noisy point can result from random subset composition

Rubric:
Full credit requires identifying class skew and random selection as the cause of the unrepresentative subset, noting the absence of positive examples, and connecting that absence to both difficult learning and a noisy curve value.

Diagnose a noisy learning-curve point from a ten-example subset.

Question:
Answer in one to three sentences.

Sample answer:
A particularly bad small subset, such as one with many ambiguous or mislabeled examples, can produce a worse-than-expected learning-curve value. A particularly good subset can produce a better-than-expected value, so the curve fluctuates at small training-set sizes.

Key points:
- Bad subsets can yield worse-than-expected values
- Good subsets can yield better-than-expected values
- Random variation is more visible at small training-set sizes

Rubric:
The answer should contrast the effects of unusually good and bad subsets and connect both effects to fluctuation in learning-curve values.

Learn Before

Related