Once a team is aligned on the evaluation metric to optimize, they will be able to make faster progress.

Google

When multiple evaluation metrics matter, such as accuracy and running time of a learning algorithm, one way to combine them is to define an acceptable value for one metric and then optimize another metric subject to meeting that criterion. In the passage's example, running time is the satisficing metric and accuracy is the optimizing metric.

Optimizing and Satisficing Metrics

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

A satisficing metric only has to be good enough by meeting a chosen value. For example, if running in 100ms is acceptable, then running time can be used as a satisficing metric that should take at most 100ms.

Satisficing Metric

An optimizing metric is the evaluation metric to optimize after the satisficing metric criteria have been set. In the passage's classifier example, accuracy is the optimizing metric when accuracy is maximized subject to meeting the running time criterion.

Optimizing Metric

When trading off N different criteria, such as a model's binary file size, running time, and accuracy, one approach is to set N-1 criteria as satisficing metrics that must meet certain values, and define the final criterion as the optimizing metric. For example, one can set acceptable thresholds for binary file size and running time, then optimize accuracy given those constraints.

Choosing N-1 Satisficing Metrics and One Optimizing Metric

For a hardware device that uses a microphone to listen for a particular wakeword, both false positive rate and false negative rate matter. One reasonable performance goal is to minimize the false negative rate as the optimizing metric, subject to no more than one false positive every 24 hours of operation as the satisficing metric.

Learn Before

Related