One could test whether a change to an algorithm makes a statistically significant difference on the dev set. In practice, most teams do not do this unless they are publishing academic research papers, and the author usually does not find statistical significance tests useful for measuring interim progress.

Google

A dev set should be large enough to detect differences between the algorithms being tried. For example, a 100-example dev set would not be able to detect a 0.1 percentage-point accuracy difference such as 90.0% versus 90.1%. Dev sets of 1,000 to 10,000 examples are common, and a dev set could be much larger than 10,000 when teams want to detect even smaller improvements.

Sizing the Dev Set to Detect Meaningful Accuracy Changes

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

A dev set with 100 examples would not be able to detect a 0.1 percentage-point accuracy difference, such as 90.0% versus 90.1%. Compared to other machine learning problems the author has seen, a 100-example dev set is small.

Small Dev Sets Cannot Detect Tiny Accuracy Differences

Dev sets with 1,000 to 10,000 examples are common. With 10,000 examples, there is a good chance of detecting an improvement of 0.1%.

Common Dev Set Sizes for Detecting 0.1% Improvements

For mature and important applications such as advertising, web search, and product recommendations, teams may be highly motivated to find even a 0.01% improvement because it directly affects company profits. In that case, the dev set could be much larger than 10,000 examples to detect even smaller improvements.

Learn Before

Related