Case Study

Decide whether a team has used the correct criterion for test-set size.

Case context: A team selected a test set but cannot say whether it gives high confidence in the system's overall performance. They argue that choosing any test-set size completes the task.

Question: Diagnose the team's mistake and state what it should require before treating the test set as sufficiently large.

Sample answer: The team has not applied the source's criterion. Merely choosing a size is insufficient; the test set should be large enough to give high confidence in the system's overall performance.

Key points:

  • Choosing any size does not establish adequacy.
  • Adequacy requires high confidence.
  • Confidence must concern overall system performance.
  • The test set should be large enough to meet that requirement.

Rubric: The response should identify the missing high-confidence requirement and connect it explicitly to overall system performance and adequate test-set size.

0

1

Updated 2026-07-19

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Machine Learning Strategy

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Yearning @ DeepLearning.AI