Case Study

Decide whether a team has used the right standard for test-set size.

Case context: A robotics team builds a test set for a navigation system. They say that once the test set exists, its size is automatically acceptable, even though they cannot trust the reported overall accuracy very much.

Question: Identify the flaw in their reasoning and state the condition that must hold before the test set can be considered large enough.

Sample answer: The team is missing the key requirement. A test set is not adequate merely because it was created; it must be large enough that its estimate of overall system performance is reliable with high confidence.

Key points:

  • Having any test set is not enough.
  • The size must support a reliable estimate.
  • The estimate should concern overall system performance.
  • High confidence is the deciding criterion.

Rubric: The response should mention the missing high-confidence requirement and tie it to a reliable estimate of overall system performance and adequate test-set size.

0

1

Updated 2026-08-12

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Machine Learning Strategy

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Yearning @ DeepLearning.AI