Case Study

Judge whether an uncertain evaluation result means the holdout set is too small.

Case context: A team checks a classifier with a holdout set, but the reported result still varies too much for the team to trust it.

Question: Using only the idea of evaluation-set sizing, what should the team conclude or investigate?

Sample answer: The team should conclude that the holdout set may not yet be large enough to support the level of certainty they need. The central question is whether the evaluation sample can produce a dependable estimate of the model’s eventual real-world performance.

Key points:

  • The estimate is still too uncertain.
  • The sizing goal is a dependable estimate.
  • The current holdout set has not met the required certainty level.
  • This diagnosis is about estimate reliability, not a claim that the model itself is necessarily poor.

Rubric: The response should connect the uncertainty to whether the evaluation set is sufficiently large for a dependable estimate, without treating uncertainty as proof of weak model performance.

0

1

Updated 2026-08-12

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Strategy

Machine Learning Yearning @ DeepLearning.AI