Learn Before
Decide whether a team has used the right standard for test-set size.
Case context: A robotics team builds a test set for a navigation system. They say that once the test set exists, its size is automatically acceptable, even though they cannot trust the reported overall accuracy very much.
Question: Identify the flaw in their reasoning and state the condition that must hold before the test set can be considered large enough.
Sample answer: The team is missing the key requirement. A test set is not adequate merely because it was created; it must be large enough that its estimate of overall system performance is reliable with high confidence.
Key points:
- Having any test set is not enough.
- The size must support a reliable estimate.
- The estimate should concern overall system performance.
- High confidence is the deciding criterion.
Rubric: The response should mention the missing high-confidence requirement and tie it to a reliable estimate of overall system performance and adequate test-set size.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
What is the key requirement for choosing a test set size?
A sufficiently large test set makes the final performance estimate more reliable.
A test set should support a high-confidence estimate of the model's _____ performance.
Match each test-set concept to the meaning it carries in this context.
Order the reasoning used to decide whether a test sample is large enough to support a dependable estimate.
Choosing test-set size for a reliable estimate
Decide whether a team has used the right standard for test-set size.
Why should the test set be made large enough?
Why test-set size matters
Increasing the test-set size does not change how confident we can be in the estimated overall performance.