Question: In a concise analytical response, explain what the comparison between 90.0% and 90.1% accuracy shows about a dev set containing 100 examples.

Sample answer: The example shows that a 100-example dev set is too small to detect a 0.1 percentage-point accuracy difference between classifiers scoring 90.0% and 90.1%. Thus, it cannot distinguish this tiny accuracy change. The author also describes 100 examples as small compared with dev sets in other machine learning problems the author has seen.

Key points:

• The dev set contains 100 examples.
• The classifier accuracies are 90.0% and 90.1%.
• The difference is 0.1 percentage point.
• The 100-example dev set cannot detect this difference.
• The author considers 100 examples small compared with other machine learning problems observed.

Rubric: A strong response identifies the 100-example dev set, the 90.0% versus 90.1% comparison, the 0.1 percentage-point difference, and the conclusion that this difference cannot be detected. It should also accurately state the author's comparison with other machine learning problems.