Explain why the learning curve is jagged for a skewed classifier and choose an appropriate subset strategy.
Case context: A team is training a model to detect machine failures from sensor logs. Only 10% of the training records correspond to failures. To build a learning curve, the team draws random training subsets of sizes 20, 50, 100, and 500 from the full training set. The resulting curve is very uneven, especially for the smallest subsets.
Question: Why is the curve so noisy at small subset sizes, and what sampling change should the team make? Under your proposed change, how many failure examples should a subset of size 20 contain?
Sample answer: The curve is noisy because random sampling from a dataset with only 10% failure cases causes large variation in the class mix for small subsets. With only 20 examples, the subset might accidentally contain too few or too many failures, which changes training behavior a lot from one draw to another. The team should build class-balanced subsets that preserve the original class proportions. For a subset of 20, 10% should be failures, so it should contain exactly 2 failure examples and 18 normal examples.
Key points:
- Random small subsets from imbalanced data can have unstable class counts.
- That instability makes the learning curve jagged.
- Use subsets that match the original class proportions.
- A subset of 20 should contain 2 failure examples at a 10% rate.
Rubric: The user must explain the noise as a consequence of high variance in class composition from random sampling, recommend matched-proportion subsets, and correctly calculate that 10% of 20 is 2.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
In a fraud-detection learning curve, why might a class-balanced subsample be better than a purely random subsample when the data are very skewed?
True or False: On highly imbalanced data with many labels, drawing ordinary random subsets usually makes learning curves less noisy than building subsets that preserve the class mix.
When you downsample a dataset with many classes and uneven frequencies, a _____ subset preserves the class mix from the original data as closely as practical.
When is a balanced subset useful for learning-curve plots?
A class-balanced subset for a learning curve should keep class proportions close to those in the full training set.
Reducing curve noise with a representative sample
Match each sampling idea with its description.
Put the steps in order for building one class-balanced sample used to estimate a point on a learning curve.
Why can balanced subsets be useful when drawing learning curves for highly skewed or many-class data?
True or False: Randomly drawing very small training subsets always gives smooth learning curves, even when some classes are rare.
In a 12-example subset that keeps the same 25% positive rate as the full data, you should include _____ positive examples.
Match each data setup to the sampling issue it creates in small learning-curve subsets.
Order the steps for deciding whether to use stratified subsets when drawing learning-curve data.
Why do skewed class distributions make learning curves noisy, and how do balanced subsets help?
Explain why the learning curve is jagged for a skewed classifier and choose an appropriate subset strategy.
How should class proportions be chosen for balanced learning-curve samples?