Google

When training many models for a learning curve is expensive, it is often enough to test a handful of unevenly spaced dataset sizes. For example, using 500, 1,000, 2,000, 5,000, and 8,000 examples can reveal the overall shape of performance while requiring fewer full training runs than checking every equal step.

Using Uneven Training Sizes to Save Time on Learning Curves

Why sample training set sizes at uneven intervals?

Smaller Training Sets Usually Train Faster

Reducing Cost with Nonlinear Data Sizes

Choosing Dataset Sizes for Learning Curves

Choosing Spacing for Learning-Curve Experiments

Question:
Explain why learning curves are often measured using training set sizes that are spaced unevenly. Describe what makes this faster and when the idea is worth using.

Sample answer:
Uneven spacing is used to reduce the amount of computation needed to build a learning curve. Small training runs finish quickly, but once the dataset becomes large, each additional model takes much longer to train. If you evaluate sizes in equal steps, you end up spending time on many expensive large runs that add only a little new information. Using a sequence that grows more quickly, such as 500, 1,000, 2,000, 4,000, and 8,000 examples, still shows the overall shape of the curve while avoiding some of the slow intermediate experiments. This approach is most useful when training each extra model is costly; if models train quickly, simple equal spacing is usually acceptable.

Key points:
- The main goal is to cut down total computation.
- Larger training sets take longer to train than smaller ones.
- Faster-growing size choices still reveal the overall curve.
- The method matters most when extra runs are expensive.

Rubric:
The essay should identify reduced computation as the main reason for uneven spacing. It should explain that larger runs take longer, that skipping some intermediate large sizes saves time without losing the general trend, and that the method is only worthwhile when training cost is substantial.

Why Uneven Training Sizes Save Time

Case context:
You are leading a machine learning effort in which training a model on 12,000 examples takes 9 hours. A new team member suggests building a learning curve by fitting models on 1,500, 3,000, 4,500, 6,000, 7,500, 9,000, 10,500, and 12,000 examples.

Question:
What problem do you see with this plan, and what different set of training-set sizes would better fit a cost-conscious learning-curve study?

Sample answer:
I would point out that this approach wastes a lot of compute, because it keeps retraining on many larger datasets that are each slow to process. A better choice is to use a smaller number of unevenly spaced training-set sizes, such as 1,500, 3,000, 6,000, and 12,000 examples. That keeps the experiment much cheaper while still revealing the overall shape of performance as the data size grows.

Key points:
- Identify the plan as unnecessarily expensive to run
- Recommend unevenly spaced training-set sizes instead of many evenly spaced ones
- Explain that the smaller set of runs still shows the main curve behavior
- Explain that the revised plan avoids repeated work on several costly large-data training jobs

Rubric:
The response must notice that the proposed evenly spaced schedule is too costly and suggest a nonlinearly spaced alternative. It should explain that the alternative reduces the number of expensive training runs while still preserving the important trend information from the learning curve.

Selecting Sample Sizes for an Efficient Learning Curve

Question:
Under what condition is it worth choosing training-set sizes that are spaced unevenly instead of evenly?

Sample answer:
This is useful when training every additional model for a standard evenly spaced learning curve would take a lot of compute time or money. Using fewer, strategically chosen sizes can reduce the cost of the experiment.

Key points:
- The method is chosen when training cost is high.
- It helps avoid spending too much compute on many extra models.

Rubric:
The answer must state that uneven spacing is relevant when the cost of training the extra models is significant.

Learn Before

Related