The Precision of a cat classifier is the fraction of images in the dev or test set that it labeled as cats that really are cats.

Precision (Cat Classifier)

The Recall of a cat classifier is the percentage of all cat images in the dev or test set that it correctly labeled as a cat.

Recall (Cat Classifier)

If both Precision and Recall matter, one standard way to combine them into a single number is to take the average of precision and recall.

Combining Precision and Recall into a Single Number

If the accuracy of a cat classifier is tracked separately in four key markets, this gives four metrics. By taking an average or weighted average of these four numbers, one ends up with a single number metric. Taking an average or weighted average is one of the most common ways to combine multiple metrics into one.

Combining Multiple Metrics via Averaging

Without a specific dev set and metric, a team may have to incorporate a new classifier into an app and manually try the app to judge whether the classifier is an improvement. Having a dev set and metric allows the team to quickly detect which ideas are giving small or large improvements, and therefore quickly decide which ideas to keep refining and which ones to discard.

Detecting Small Performance Improvements with a Dev Set and Metric

During development, a team will try many ideas about algorithm architecture, model parameters, and choice of features. Having a single-number evaluation metric such as accuracy allows the team to sort models according to their performance on the metric and quickly decide what is working best. It also speeds up decisions when selecting among a large number of classifiers by giving a clear preference ranking and a clear direction for progress.

Google

Once a dev set and test set are defined, a team can try many ideas, such as different learning algorithm parameters, to see what works best. The dev and test sets allow the team to quickly see how well the algorithm is doing, and their purpose is to direct the team toward the most important changes to make to the machine learning system. Measuring each idea's performance on the dev set lets the team quickly decide whether they are heading in the right direction, and helps them identify which ideas to keep refining and which ones to discard.

Purpose of Dev and Test Sets

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Single-Number Evaluation Metric

Having an initial dev/test set and metric helps a team iterate quickly.

Initial Dev/Test Set and Metric Helps Quick Iteration

If the initial dev/test set or metric missed the mark, change it quickly. One warning sign is that the dev set plus metric ranks one classifier higher while the team thinks another classifier is superior for the product. Possible causes include a dev/test set distribution that differs from the actual distribution of interest, overfitting to the dev set, or a metric that measures something other than what the project needs to optimize. After changing dev/test sets or metrics, make sure the team knows the new direction.

Changing Dev/Test Sets or the Metric When They No Longer Guide the Team

Error analysis uses data about an ML system?s mistakes to derive insight about what to do next. There is no single right way to carry it out; common design patterns help, but teams can experiment with other ways of analyzing errors.

Error Analysis

When deciding to improve label quality, consider double-checking the labels of both examples that the system misclassified and examples it correctly classified. It is possible that both the original label and the learning algorithm were wrong on an example.

Learn Before

Related

Learn After