Google

A dev set with 100 examples would not be able to detect a 0.1 percentage-point accuracy difference, such as 90.0% versus 90.1%. Compared to other machine learning problems the author has seen, a 100-example dev set is small.

Small Dev Sets Cannot Detect Tiny Accuracy Differences

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

What can a 100-example dev set reveal about 90.0% versus 90.1% accuracy?

Can 100 dev examples detect a 0.1 percentage-point accuracy difference?

A 100-example dev set cannot detect a _____ percentage-point accuracy difference.

Match each element of the example to its source-based description.

Order the reasoning used to assess the 100-example dev set.

Question:
In a concise analytical response, explain what the comparison between 90.0% and 90.1% accuracy shows about a dev set containing 100 examples.

Sample answer:
The example shows that a 100-example dev set is too small to detect a 0.1 percentage-point accuracy difference between classifiers scoring 90.0% and 90.1%. Thus, it cannot distinguish this tiny accuracy change. The author also describes 100 examples as small compared with dev sets in other machine learning problems the author has seen.

Key points:
- The dev set contains 100 examples.
- The classifier accuracies are 90.0% and 90.1%.
- The difference is 0.1 percentage point.
- The 100-example dev set cannot detect this difference.
- The author considers 100 examples small compared with other machine learning problems observed.

Rubric:
A strong response identifies the 100-example dev set, the 90.0% versus 90.1% comparison, the 0.1 percentage-point difference, and the conclusion that this difference cannot be detected. It should also accurately state the author's comparison with other machine learning problems.

Explain why the example illustrates a limitation of a small dev set.

Case context:
A team compares classifier A at 90.0% accuracy with classifier B at 90.1% accuracy using a dev set of 100 examples.

Question:
Based only on the source, what should the team conclude about its ability to detect the classifiers' accuracy difference?

Sample answer:
The team should conclude that its 100-example dev set cannot detect the 0.1 percentage-point difference between the classifiers. The source characterizes a 100-example dev set as small compared with other machine learning problems the author has seen.

Key points:
- The accuracies differ by 0.1 percentage point.
- The dev set has 100 examples.
- The dev set cannot detect the stated difference.
- A 100-example dev set is described as small.

Rubric:
The response should reject the claim that the dev set can distinguish the classifiers, identify the difference as 0.1 percentage point, and connect the limitation to the 100-example dev-set size.

Diagnose whether a small dev set can distinguish two nearly equal classifiers.

Question:
Answer in one to three sentences using the classifier accuracies and the stated limitation.

Sample answer:
The dev set is inadequate because 100 examples cannot detect the 0.1 percentage-point difference between 90.0% and 90.1% accuracy. The author considers a 100-example dev set small compared with other machine learning problems observed.

Key points:
- 100-example dev set
- 90.0% versus 90.1% accuracy
- The difference cannot be detected

Rubric:
The answer should mention the 100-example size, the 0.1 percentage-point difference or both accuracies, and the inability to detect that difference.

Learn Before

Related