Diagnose whether a small dev set can distinguish two nearly equal classifiers.
Case context: A team compares classifier A at 90.0% accuracy with classifier B at 90.1% accuracy using a dev set of 100 examples.
Question: Based only on the source, what should the team conclude about its ability to detect the classifiers' accuracy difference?
Sample answer: The team should conclude that its 100-example dev set cannot detect the 0.1 percentage-point difference between the classifiers. The source characterizes a 100-example dev set as small compared with other machine learning problems the author has seen.
Key points:
- The accuracies differ by 0.1 percentage point.
- The dev set has 100 examples.
- The dev set cannot detect the stated difference.
- A 100-example dev set is described as small.
Rubric: The response should reject the claim that the dev set can distinguish the classifiers, identify the difference as 0.1 percentage point, and connect the limitation to the 100-example dev-set size.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
What can a 100-example dev set reveal about 90.0% versus 90.1% accuracy?
Can 100 dev examples detect a 0.1 percentage-point accuracy difference?
A 100-example dev set cannot detect a _____ percentage-point accuracy difference.
Match each element of the example to its source-based description.
Order the reasoning used to assess the 100-example dev set.
Explain why the example illustrates a limitation of a small dev set.
Diagnose whether a small dev set can distinguish two nearly equal classifiers.
Why is the 100-example dev set inadequate in the source example?
Which observation best supports calling the dev set too small for this comparison?
Does the author characterize 100 dev examples as small?