If learning-curve noise makes the true trend hard to see, one remedy is to sample several different training sets of the same small size, train a different model on each, compute each model's training and dev error, and plot the average training and dev error.

Averaging Learning Curves over Multiple Random Subsets

When training data is skewed toward one class or has many classes, learning curves can be less noisy if each small subset is balanced so that class fractions are as close as possible to the overall fractions in the original training set.

Balanced Subsets for Noisy Learning Curves in Skewed or Many-Class Data

When the training set is small, learning-curve points can fluctuate because randomly chosen small subsets may be unusually good or bad, such as containing many ambiguous or mislabeled examples. Skewed class distributions or many-class problems increase the chance of selecting an unrepresentative small subset.

Google

A learning curve plots a learning algorithm's dev-set error against the number of training examples.

Learning Curve for Dev-Set Error

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

To plot a learning curve, run the algorithm using different training-set sizes and then plot how dev-set error varies with training-set size. For example, with 1,000 examples, separate copies of the algorithm might be trained on 100, 200, 300, and more examples up to 1,000.

Constructing a Learning Curve by Varying Training Set Size

As training-set size increases, dev-set error should decrease.

Dev-Set Error Should Decrease as Training Set Size Increases

A desired error rate is a level of error one hopes a learning algorithm will eventually achieve. The desired level of performance can be added to a learning curve.

Desired Error Rate for a Learning Algorithm

After adding the desired performance level to a learning curve, visually extrapolating the dev-error curve can help guess how much closer adding more data could get to the desired level. In the passage's example, doubling the training-set size looked plausibly sufficient to reach the desired performance.

Using a Dev-Error Learning Curve to Estimate the Benefit of More Data

If one only looks at the dev-error curve, it can be hard to extrapolate and predict exactly where the curve will go with more data. Training error is an additional plot that can help estimate the impact of adding more data.

Training Error Plot for Estimating the Effect of More Data

Plotting both training error and dev error on the same learning curve makes it easier to extrapolate dev-error behavior. Looking at the full curve gives a more comprehensive picture of algorithm performance across different training-set sizes than measuring training and dev error only at the largest available training set size.

Learn Before

Related

Learn After