Question: In a concise analytical response, explain the evidence-based decision rule for averaging over multiple subsets or constructing balanced subsets, including the role of dataset size and class distribution.

Sample answer: First plot the learning curves and inspect whether their noise hides the underlying trends. Averaging over multiple subsets or constructing balanced subsets is worth the effort only if that problem is actually observed. If the training set is large, such as over 10,000 examples, and its class distribution is not very skewed, the techniques probably will not be needed.

Key points:

• Plot the learning curves first.
• Check whether noise hides the underlying trends.
• Consider averaging or balanced subsets only after excessive noise is observed.
• A training set over 10,000 examples with a class distribution that is not very skewed probably does not require these techniques.

Rubric: A strong response states that the initial curves must be plotted first, identifies excessive noise that obscures trends as the trigger, names the two relevant techniques, and explains why a large dataset with a distribution that is not very skewed reduces their likely usefulness.