Diagnose whether a team's uncertain final-performance estimate signals a test-set sizing problem.
Case context: A team uses its test set to estimate the final performance of a system, but the team does not feel confident in the estimate.
Question: Based only on the test-set sizing principle, what should the team diagnose or decide?
Sample answer: The team should diagnose that the test set has not yet demonstrated that it is big enough for its purpose. The relevant decision criterion is whether the test set supports a confident estimate of the system's final performance.
Key points:
- The current estimate is not confident.
- A confident final-performance estimate is the sizing objective.
- The test set has not satisfied the stated criterion for being big enough.
- The diagnosis concerns estimation confidence, not necessarily performance quality.
Rubric: The response should connect the lack of confidence directly to the test set's sizing criterion without claiming that the system's performance itself is necessarily poor.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What should determine whether a test set is large enough?
A test set is adequately sized when it supports a confident estimate of final system performance.
Complete the principle: The test set should support a _____ estimate of final performance.
Match each part of the test-set sizing principle with its role.
Order the reasoning used to judge whether a test set is sufficiently large.
Explain how confidence in final-performance estimation defines a sufficiently large test set.
Diagnose whether a team's uncertain final-performance estimate signals a test-set sizing problem.
Why must a test set be big enough?
Which team statement best applies the test-set sizing principle?
Test-set size should be judged independently of confidence in the final-performance estimate.