When Is a Test Set Large Enough?
Question: In a concise analytical response, describe how test-set size, confidence, and final-performance estimation are connected.
Sample answer: A test set is large enough when it supports a dependable estimate of how the final system will perform. The point is not to maximize the number of examples for its own sake. Instead, the test set should be big enough that the estimated performance can be trusted with the desired level of confidence.
Key points:
- The quantity being estimated is the system’s final performance.
- The estimate must be reliable enough to inspire confidence.
- Test-set size is adequate only if it supports that confidence.
- The needed size depends on the estimation goal, not on a fixed rule.
Rubric: A strong response states that the test set must be big enough to produce a confident estimate of final system performance and makes clear that adequacy is determined by that estimation purpose.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What makes a test set large enough?
A holdout test set is large enough when it can give a dependable estimate of how the final model will perform.
How Test Set Size Supports Evaluation
Match each test-split idea to its meaning.
Order the steps for deciding whether an evaluation set is large enough.
When Is a Test Set Large Enough?
Judge whether an uncertain evaluation result means the holdout set is too small.
Why should an evaluation set contain enough examples?
Which statement best reflects how to choose a test-set size?
Test-set size can be chosen without regard to desired confidence.