Primary Benefit of Initial Dev/Test Set and Metric

Dev/Test Set and Metric Impact on Iteration Speed

Benefit of Initial Dev/Test Set and Metric _____

Matching Dev/Test Components to Outcomes

Sequence for Establishing a Quick-Iteration Workflow

Facilitating Quick Iteration in Machine Learning

Rapid Iteration and Initial Setup

Initial Dev/Test Set and Metric _____ Quick Iteration

Matching Quick Iteration Concepts

Decision Path for Achieving Quick Iteration

Having an initial dev/test set and metric helps a team iterate quickly.

Google

Once a dev set and test set are defined, a team can try many ideas, such as different learning algorithm parameters, to see what works best. The dev and test sets allow the team to quickly see how well the algorithm is doing, and their purpose is to direct the team toward the most important changes to make to the machine learning system. Measuring each idea's performance on the dev set lets the team quickly decide whether they are heading in the right direction, and helps them identify which ideas to keep refining and which ones to discard.

Purpose of Dev and Test Sets

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

During development, a team will try many ideas about algorithm architecture, model parameters, and choice of features. Having a single-number evaluation metric such as accuracy allows the team to sort models according to their performance on the metric and quickly decide what is working best. It also speeds up decisions when selecting among a large number of classifiers by giving a clear preference ranking and a clear direction for progress.

Single-Number Evaluation Metric

Initial Dev/Test Set and Metric Helps Quick Iteration

If the initial dev/test set or metric missed the mark, change it quickly. One warning sign is that the dev set plus metric ranks one classifier higher while the team thinks another classifier is superior for the product. Possible causes include a dev/test set distribution that differs from the actual distribution of interest, overfitting to the dev set, or a metric that measures something other than what the project needs to optimize. After changing dev/test sets or metrics, make sure the team knows the new direction.

Changing Dev/Test Sets or the Metric When They No Longer Guide the Team

Error analysis uses data about an ML system?s mistakes to derive insight about what to do next. There is no single right way to carry it out; common design patterns help, but teams can experiment with other ways of analyzing errors.

Error Analysis

When deciding to improve label quality, consider double-checking the labels of both examples that the system misclassified and examples it correctly classified. It is possible that both the original label and the learning algorithm were wrong on an example.

Learn Before

Related

Learn After