Case Study

Diagnose whether a team's uncertain final-performance estimate signals a test-set sizing problem.

Case context: A team uses its test set to estimate the final performance of a system, but the team does not feel confident in the estimate.

Question: Based only on the test-set sizing principle, what should the team diagnose or decide?

Sample answer: The team should diagnose that the test set has not yet demonstrated that it is big enough for its purpose. The relevant decision criterion is whether the test set supports a confident estimate of the system's final performance.

Key points:

  • The current estimate is not confident.
  • A confident final-performance estimate is the sizing objective.
  • The test set has not satisfied the stated criterion for being big enough.
  • The diagnosis concerns estimation confidence, not necessarily performance quality.

Rubric: The response should connect the lack of confidence directly to the test set's sizing criterion without claiming that the system's performance itself is necessarily poor.

0

1

Updated 2026-07-19

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Strategy

Machine Learning Yearning @ DeepLearning.AI