Case Study

Diagnose whether a small dev set can distinguish two nearly equal classifiers.

Case context: A team compares classifier A at 90.0% accuracy with classifier B at 90.1% accuracy using a dev set of 100 examples.

Question: Based only on the source, what should the team conclude about its ability to detect the classifiers' accuracy difference?

Sample answer: The team should conclude that its 100-example dev set cannot detect the 0.1 percentage-point difference between the classifiers. The source characterizes a 100-example dev set as small compared with other machine learning problems the author has seen.

Key points:

  • The accuracies differ by 0.1 percentage point.
  • The dev set has 100 examples.
  • The dev set cannot detect the stated difference.
  • A 100-example dev set is described as small.

Rubric: The response should reject the claim that the dev set can distinguish the classifiers, identify the difference as 0.1 percentage point, and connect the limitation to the 100-example dev-set size.

0

1

Updated 2026-07-19

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Machine Learning Strategy

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Yearning @ DeepLearning.AI